Understanding and Dealing with Click Fraud

November 13th, 2010
Comments Off

This is an excerpt from a whitepaper that I was commissioned to write for the FBI. It was part of an internal research project where they were to determine the effects and solutions regarding the rampant Click Fraud concern with online advertising and the like. – Brian Grayless

What is click fraud?

Let’s start out by defining some key terms that are important in understanding click fraud.

  • Advertiser – The entity that pays money to get traffic to their site in the way of bidding on keywords or topical categories (bid auctions).
  • Publisher – Any entity which displays advertiser ads on their web site or in some other publicly viewable medium.
  • Visitor – A legitimate user who clicked on something to get to the appropriate target web site.
  • Click – A visitor to the advertiser’s site that came by route of one or more publishers.
  • PPC (Pay Per Click) – An internet advertising model where the amount the advertiser pays is dictated on a per click basis for the terms (keyword or categories) being bid on.
  • CPC (Cost Per Click) – The amount (bid price) paid by the advertiser to receive one visitor for a particular term. The amount is paid only if the visit occurs.
  • CPA (Cost Per Action) – An advertising cost associated to a particular desired visitor action, i.e. purchased a product or service, filled out a survey, or signed up for a newsletter.
  • Conversion – A completion of the advertiser’s desired action under a CPA advertising model.
  • Click Stream – The route the click traffic takes from the time the click is made through the time the web user arrives on the advertiser’s targeted URL. There can often be URL redirects and several publishers (usually tracked by cookies or ID’s in the URL) that receive information for each click, completely transparent to the user.
  • Rev Share – A single publisher’s fraction of the revenue generated by specific click and conversion sources. For example, a smaller publisher might arrange to send click traffic into a larger publisher’s click stream, providing the larger publisher with more traffic and retaining a 5% rev share of the total per click amount for the smaller publisher. Rev Share can be seen as a multi-tier sales commission.
  • Ad Feed – ad listings/data provided by an n-tier publisher by request to display to users on another publisher’s web site or application.

Click fraud, generally speaking, occurs when something (person, web bot, etc…) posing as a legitimate internet user follows (or clicks) a paid advertisement URL to the advertisers web site from which money is generated for some entity other than the advertiser.

Valid User

An advertiser pays good money for advertising, expecting that a portion of the traffic received in return will generate revenue in some fashion. Non-legitimate visitors produce bad clicks which in effect spend advertiser dollars with no hope of a return for the advertiser. This expense is instead divvied up between the layers (rev share) of publishers that are likely to be present in the click stream. Publishers, especially the ones on the end of the chain, often have the most to gain from this practice and will devise all sorts of innovative ways to game the system. Larger publishers in the click stream will often ignore or downplay this activity, knowing that it lines their pockets in the process.

In short, advertisers are being hijacked of their advertising dollars from inflated term bidding marketplaces because of traffic that is posing as real, live, interested web site visitors. It is theft akin to diverting fractions of a penny from financial transactions to a private account.

See more information on Click Fraud and get the complete and detailed whitepaper.

Development , , , , , , , ,

A New Vision for Success

July 23rd, 2010
Comments Off

SuccessThe desire for “success” is an integral part of the American Dream. College graduates everywhere bask in the glory of their newfound knowledge, in hopes that their education will lead them to be successful in their field, in their lives, and in their homes. We tend to measure success by how much we have, yet that same surface-level measure for success is also at the root of the recent economic crisis. We Americans have a lot, don’t we? And it never seems to be enough. Nice homes, decent salaries, unlimited forms of entertainment at our finger tips. Yet at the end of the day, we can’t even keep the clothes on our backs because we gave every asset we had to get the end result… “stuff”. We sold ourselves short trying to live the Great American Dream. I don’t think this is what the first Americans had in mind when they started dreaming.

We do the same thing to our businesses. We have this picture in our head of what our job or business looks like. We do our best to educate ourselves and acquire the experience necessary to reach success. However, we are selling ourselves short. We want success and we want it now! We don’t have time to properly plan and prioritize… we have a business to run. We don’t value the feedback and abilities of our people… after all “It’s our business. If an employee wants to have an opinion, they can start their own. We didn’t hire them to be talented. We hired them to do a job.”

Does some of this sound familiar? Maybe you’ve heard similar sentiments from your managers. Perhaps this comes across in unspoken communication throughout your organization. Perhaps YOU even think this way but have a hard time admitting it because it sounds so ruthless… so insensitive… so careless. In the quiet corridors of our minds, most of us see the flaws in this way of thinking. Yet, we go throughout our daily routine, getting sucked into this vision of success that has been pounded into our corporate cultures… this idea that “business is business”… that the quickest way there is the best way. As a result of this “do or die” attitude, we never take the time to see the symptoms of our own condition. We recognize that there are some issues that need addressing – the same issues that our competitors deal with – and figure that it’s just a natural part of doing business. After all, high turnover, low morale, shoddy production, employee backbiting and disrespect are normal issues for a growing company, right?

Just like our desire to live the American Dream, we sit in our comfy cubicles and glass offices and find ourselves in a situation that we did not expect. We meant well, but we eventually come to terms with the reality that true success is still as far away as it ever was.

Hopefully, this doesn’t describe you. Perhaps you are more astute than the common professional, or maybe you’ve worked in the trenches and see the reality I’m describing. You may recognize that change is needed, but what?

A common theme among business analysts and authors is that many companies fail due to internal problems, not market changes or supply and demand, etc… They simply implode due to various factors that get out of control over time and lead to lost productivity, poor morale and inferior products and services. These kinds of problems are often highlighted when referring to the differences between two relatively similar companies where one is wildly successful and the other simply fades into the distance or crashes abruptly. After months of taking shortcuts, many companies find that all the capital, marketing and restructuring cannot save them. There is no shortcut to success.

What is needed is a core shift in the mentality behind what makes a company TRULY successful. This requires a drastic change toward more traditional values with the ability to integrate the necessary modern methodologies and tactics that are crucial to survive in a modern world and economy. A shift of this magnitude takes away the “business is business” excuse. It requires much higher levels of mutual respect among coworkers and associates. It allows producers to take pride in their work once again. When the final checklist of accomplishments is being reviewed at the quarterly board meeting, and the items in question are: Did we meet our clients expectations? Did we have positive growth? Did we focus on the right things? Are the employees satisfied and feeling a sense of contribution? Is our turnover lower?… the answer can become be a resounding “YES!”

Whether you are someone’s subordinate or in charge of the whole enchilada, the difference can be made starting today. The crux of the larger problem comes down to how you, and perhaps all the people in your organization, perceive your business… your paradigm. But what about this paradigm is so ineffective? I would imagine that your organization, like most others, puts the majority of its resources into things like: improving the bottom line, meeting deadlines. While these facets of business definitely have a place in reaching and sustaining success, they too often become the only facets that receive any attention. This is due in part to their instant measurability. You always know what your profit looks like. You can quickly determine if you are behind a deadline. We tend to think that what we can easily measure is all there is. And so our progress, success and even failure is measured by these important, but misplaced objectives. Meanwhile, other worthwhile objectives such as longevity, employee growth and client satisfaction get completely ignored. After all, if it can’t be measured and translated into instant profit for the shareholders, then it doesn’t serve a purpose.

This “I want it now!” perspective is common with many other things as well. Various industries and the general population ignore pressing issues of pollution and natural resources, enjoying what the earth has to offer today, without much concern for the future. Essentially, short term demand almost always dominates long term progress and sustainability. It’s a very selfish, impatient and shortsighted perspective.

Other more important factors such as people, culture, future and sustainability are overlooked because they aren’t tangible. They can’t be calculated. They don’t fit nicely on a graph or quarterly report. How do you quantify the quality or satisfaction of your people? How do you measure the nuance of a company culture? How do you determine what your future looks like or whether or not you are moving in a sustainable direction? You can’t really… which is why there will always be organizations whose focus is on nothing but the numbers. Bottom line? Numbers. Products? Numbers. Employees? Numbers. Layoffs? Numbers. Of course, you can do a slew of surveys, create review committees and hire analysts dedicated to the purpose of quantifying the intangible. However, this just feeds back into the bureaucratic measurability of what can be measured and understood. You’ll never have the full picture.

What is needed are people that can, or can be taught to, appreciate the human factors of an organization. People that are willing to take a chance on other people… by letting Bob off a bit early to go take care of his sick kids… by allowing the team to set their own goals, deadlines and expectations (I know. It sounds absolutely ludicrous to have a professional actually know what they are doing.)… in hopes that those people will enjoy their jobs more, feel respected, gain more personal balance and reward the whole of the company with their continued hard work, best efforts and hidden talents. Those with authority can exercise a bit less of it and focus on leadership. They can loosen the reigns a little and allow people to express their talents and interests. They can be willing to encourage an environment of trust, integrity, interpersonal influence and personal growth. People that can pioneer in this direction will be your indicators. They will have the pulse of the organization and will be your measurement. They will have the hunches from which effective decisions can be made.

The kind of choices necessary to successfully and effectively run a business require patience, contemplation and thought for the future impact those choices will have on the organization as a whole. How the business is run, how the projects are planned and how client expectations are set and managed come down to what the decision makers see as important. Long term, authentic, success in business comes down to a focus on priority.

What are your priorities and how do you think your focus affects your success?

Perspective , , , , ,

Tinkerer of the Intangible

June 23rd, 2010
Comments Off

You know, I’ve had a fun and interesting career thus far. I’ve worked for some great companies and for some duds, but regardless of who I’ve worked for and what I did for them, I’ve always managed to learn a lot. Not just about my work, or the industry, but about the people, the processes, the organization, the obscure or the obvious political structure or agendas, what works and what doesn’t, what motivates and what suffocates, what incubates change and what stifles it. As a matter of fact, I’ve come to recognize in myself that the most fascinating aspects of my past work experience, the most frustrating, the thoughts that keep me up at night, usually have little or nothing to do with my work assignments or specific profession. They are usually about the people, the organizational behavior or the dynamics of a thriving or failing company. There is something in me that is always acutely aware of the dynamic environment that surrounds me.

I’m a tinkerer. I mean, I have friends and associates that tinker with cars, appliances and other devices and they do amazing things. My brother’s talent with bodywork and vehicle enhancements makes Orange County Chopper look like child’s play. While I am also very mechanically adept, and also like to make or work on things of various natures, more than anything else, I’m a tinkerer of the intangible. It’s the things that escape vision or the grasp of the hand that become my mental playground.

What keeps you up at night?


Rails Sessions Across Multiple Subdomains

May 26th, 2010
Comments Off

Okay, so I’m working on a new Rails project. Things are coming along great. Then we hit a snag where our SSL is not working as expected. We want it to work on Staging and Production only, and only for the actions that we need them on. So, the SslRequirement Gem did the trick.

However, we have many (and many more to come) sub-subdomains which caused another dilemma. We have a wildcard SSL certificate, however, although we can get one that also handles sub-subdomains, it’s not necessarily supported by the user’s browser. So, our other option was to put all the public stuff on the subdomains and have all the private stuff on a “private”.domain.com address which would adequately be handled by SSL at the application and certificate levels. After some finagling, I managed to dynamically change the subdomain based on whether or not the action requested should be SSL’d.

Everything seemed to be humming along, but this new code snippet was relying on something that we hadn’t previously tested thoroughly… sessions. Session are just supposed to work right? However, evidently they don’t work by default across subdomains. So, after some hunting around, this little snippet put into my “/config/[environment].rb” file did the trick.

config.action_controller.session = { :domain => ".[domain].com" }

Evidently, this tells the session to share across anything within the main domain. You can also restrict it further by using “.[subdomain].[domain].com”.

Works like a charm.

Ruby , ,

PHP That Just Works

September 18th, 2009
Comments Off

I’m not one of those developers that likes to waste time setting up my dev environment. If I have a project to complete, I should be coding, not messing with config files, compiling Apache or messing with PHP to load that extension I just found out I needed. I like messing with my machine but not with an impending deadline.

With all this in mind, I’ve tried to simplify my entire dev environment over the years… not from a perspective of using simple tools and sticking to basics, but instead from a perspective of optimizing my workflow and keeping my development moving. In the middle of a project, I feel that systems admin focus should be on tweaking the production machines, rather than screwing around with my local dev box.

Zend Server CE Control PanelAlways looking for ways to make development easier, I decided to give the Zend Server CE (Community Edition) a try. The idea is that it installs your PHP, Apache, MySQL (with PHPMyAdmin) and a great management console that allows you to install extensions with just a click. You can still customize our Apache conf and other things, but it works well out of the box (I’m on a Mac). While you can run it along side another Apache installation, I tweaked it to run on port 80 and handle multiple virtual hosts. While this may not be ideal for all teams, it can allow everyone to have the same environment without having to mess with poorly updated all-in-one dev environments.

One of the reasons this excites me, being on a Mac, is that every time in the past that I’ve updated my Mac OS, the install kills something on my system causing my development environment to go all wonky. Then I have to spend precious work time to fix it. The Zend Server CE install keeps everything nice and tidy and, to my knowledge, doesn’t rely on other stuff outside of the install to function (unless you are setting host entries in your hosts file, /etc/hosts on a Mac).

This, oh so sweet, environment gets a little better. While I still have a variety of development tools at my disposal, my main IDE has become Zend Studio (Eclipse). I know, I know… there are a lot of purists out there that say it’s too heavy, or sluggish, or isn’t simplistic enough. There are occasional bugs or things that annoy me, but at the end of the day it is integrated enough that it lets me get my work done. That’s the whole point of an IDE. It also integrates with Flex Builder plugin which is a plus for me.

Development, PHP, Reviews , , , , , ,

Agility Futility

April 30th, 2009
Comments Off

Agile development methods have really taken the IT world by storm. In the last few years Agile has become THE way to manage and develop software, especially among young, emerging companies. It brings to the table a flexible model for communication and progress as well as a sense of anti-corporatism which is heavily embraced in many IT workplace cultures.

While this almost hippie-ish movement of peace, love and agileness has really relaxed a lot of work cultures and has been a boon for productivity and customer interaction, there are some often ignored pitfalls which eventually leave a work culture devastated and disallusioned.

Do it for the right reasons
It’s not enough to adopt Agile just because it works well for some or because you read about it on a trade blog. Agile, or any other hastily adopted process or methodology, cannot solve all your problems. It will simply make you more of what you already are. Your weaknesses, if not already apparent, will eventually surface and you must be ready and willing to acknowledge and address them.

Successfully adopting any methodology like this requires that you have an adequate paradigm about people, business and clients which instills respect and integrity and is in sync with the methodology. If your efforts are only surface-level rhetoric, and no paradigm shift occurs, the process will fail and you’ll be looking for the next “great thing” to fix your woes.

Use best practices
While Agile lends itself to a more rapid pace of development, it can be easy to leave crucial parts of the SDLC out of the equation in the interest of time. Adequate quality assurance and testing are often the first to go. Test-driven development, which utilizes a testing process as part of your development, is a great way to minimize QA overhead while maintaining work quality. Building test code as, or before you develop may add a little to your initial timeline but will result in fewer deployment panics and provide built-in specifications for your code to adhere to.

Don’t sacrifice quality
Cutting corners is a big no-no. Decide what are features and what are bugs. Determine which of them are in your critical path and develop them properly. If you can’t do them right, choose not to do them or arrange for more time to complete the project. NO ONE benefits from poorly thought out, shoddy work. Management only seems happy until they realize the problem they rushed you to fix ends up worse than before. It is the developer’s job to speak up and communicate risks and issues which then translate into proper timeline and feature negotiation.

Don’t ignore problems. Moderately plan for the future and proactively address problems and improvements through iterations. Ignored problems build up over time and eventually result in a complete rewrite. Iterative development can be your friend. Keep track of issues and slip some into each iteration so you can keep up with the change.

Be realistic
There is an old project management addage that explains how with every project, three factors are desired: speed, low cost, and great quality. You can pick two. Having all three is a fantasy propagated by poor sales teams. This is because any improvement in any one or two of the factors will negatively affect the third. For example, if speed is crucial, it will likely affect quality and cost. If very low cost is required, completion times will often be longer and quality will suffer. The only way to realistically improve one of the factors is to improve your effectiveness in all three of them. Attempting to use Agile development concepts to short-sightedly manipulate any of these factors is counter-productive.

Avoid burn-out
Finally, keep in mind that overworking your developers is counter-intuitive in an Agile model. With a more top-down, waterfall approach, you may get away with piling on extra hours, shoving more into a deadline and driving with a whip. Burn-out doesn’t make for solid code, good morale, communication and low turnover; all which are factors behind a well-functioning Agile machine. Utilize iterations to drive realistic deadlines and continually reassess based on top priorities to keep everyone focused on the same goal.

Development, Technology , , ,

Flash on an iPhone?

April 10th, 2009
Comments Off

Flash is not supported on the iPhone

Flash is not supported on the iPhone

As many of you know, I love Apple products. Not to the exclusion of the everything else, but I’ve got to hand it to them. They’ve created product lines that, despite the often steep cost, attract users and keep them reeled in with great user experience and phenomenal practicality.

One thing has really been bugging me though… this whole issue about Flash on an iPhone. It doesn’t look like it’s going to happen anytime soon and if it does, it will likely be a trimmed down, “do nothing” version. See the Wired.com article.

While I love my iPhone and a great part of the internet can be seen on it, it really ticks me off that they are essentially censoring my use of the internet. Not because of content appropriateness or service bandwidth issues and restrictions. They are censoring the internet on my device simply so they can keep a tight grip on the money they get from the App Store and the other paid-for downloadables that they provide. If you’re going to give me the web… give it all to me and let me decide what to view.

Sure I am missing a few fun games and other things from the world of Flash. But I am also missing access to some very important websites and data that I would like to get on my iPhone, including Adobe Flex-based reporting interfaces from work.

Apple, if you’re not going to allow Flash on the iPhone, at least give me a legit technical reason or something. Otherwise, find a way to make it work. It seems ridiculous that the “most advanced” phone device in the world, that does SO much, cannot even offer Flash. Meanwhile, other second-rate devices are, simply because they can. iPhone users pay good money for their phone and service (not to mention all the other Apple products they have also likely latched on to). It’s an insult really, that many of us professionals own these fantastic phones that we use for so many facets of our business and personal lives… but we can’t have Flash because Apple wants to take an even bigger slice of our paychecks.

I’m very curious to see how they handle this issue. I’ve loved Apple because of the flexibility, the power, the beauty of it all. But, this issue seems a bit too Micro$oft for my liking.

Reviews, Technology , , , , , ,

Something NOT New for a Change?

March 27th, 2009
Comments Off

I’m never really that surprised when I see the latest gadget on the shelves or on one of the many gadget websites that are for the express purpose of flaunting the fact that you don’t have the latest and greatest “thing”. New things are…. well old. There is always something new. There is always an “upgrade”, “new model” or something “totally revolutionary”.

Revolutionary? That word has been so watered down over the years… it almost lacks meaning altogether. Is shoving 100 gigs of ram onto a USB thumb drive revolutionary? Well, no… cool and complex, but not revolutionary.

Worm Poop in a Jug

Worm Poop in a Jug

Today, I saw something that ACTUALLY impressed me. Not in the “ooh”, “ahhhh”, “gotta have it” kinda way. It caught my attention in the overall purpose and drive behind the innovation. I’m talking about the company TerraCycle, Inc..

Okay, so I’m not typically the tree hugger type, but I consider myself conservation minded. Regardless, this is just cool. TerraCycle, Inc. takes trash, just everyday trash, and makes stuff out of it, reuses bottles and other packaging, etc… What caught my interest is that most other companies continue create new “stuff” that adds to the constant barrage of “buy this” and “get that”, while adding to the mountains of trashed iPods, disc players, spent PC’s and other trendy junk already in the landfills.

While I’m not likely to go out and buy one of their “Capri Sun” handbags or a jug of worm poop, it’s their tactic and business model that I find interesting. What if we applied this to technology? What if instead of making yet another shopping cart, blog system, CMS or code framework, we actually put effort into building on top of or improving what we already have access to? What if we could find TRULY creative ways to reuse the mounds of rejected hardware filling up our landfills for something other than sculptures and doorstops? Sure, there are several developers and companies that do that, but most don’t plain and simple.

I think perhaps there is a lot to be learned by looking at this way of doing business, not only as individuals but as an industry. As developers, can we add value to our industry and the world? Or are we filling the net with endless piles of already done cruft destined to be forgotten about next week for the latest and greatest cruft? Admittedly, every day of our career can’t be overflowing with pure innovation and meaning, but looking at TerraCycle, Inc. has me thinking about how I will spend my energy and resources in the future.

See a video about TerraCycle at CNN.

Reviews, Technology , , ,

GE Brings Minority Report to Life?

March 9th, 2009
Comments Off

Okay, well not quite, but I thought this was pretty amazing. GE “brings good things to life”, almost literally.

Tom Cruise doing virtual computing

Tom Cruise doing virtual computing

For those of you that have seen Minority Report, you know that that people have been trying to recreate that type of computing model in the real world since the movie came out. Well, it doesn’t quite exist yet, but GE may be headed in the right direction.

Tom Cruise doing virtual computing

Tom Cruise doing virtual computing

As a way to draw interest to their Smart Grid energy technology, GE has created an interactive 3D experience that is pretty startling at first. It almost seems unreal… until you realize that it is actually interacting with you. Check out a video of the guys at doppelagent.de experiencing this first hand, although you will want to try it out for yourself.

Okay, so it’s not quite Minority Report level computing. However, with the live human 3D interaction inside of a virtual, yet real, space, all done in the comfort of your browser using Adobe Flash… this is quite amazing, nonetheless. I would love to see this kind of technology take off and be available in a browser… maybe we’re not far off.

Flex, Technology , , , , ,

Variable Conflicts in JavaScript

March 8th, 2009
Comments Off

It is quite common to find yourself with a heinous JavaScript error on a page that until recently seemed to work flawlessly. Perhaps you changed your JavaScript. Maybe you included a 3rd party script or a script from another domain onto your page. Now, everything that was once peachy has turned to sour grapes!

More than likely the problem is that with all the varying scripts on the page, variables from other functions will conflict with variables in the existing code, causing failures and errors, and even worse, overwrite variable values without any notification. It can take hours to track down variables that conflict between scripts before it finally works. Some developers figure that these kinds of issues are probably just inherent in client-side web development and use that as another “reason” as to why JavaScript is inferior.

I don’t think client-side development should be looked at as inherently quirky. Sure there are some browser nuances and environment issues that you can’t control, but you can develop very robust code that works well and adequately serves its purpose.

There are a few key things that you can do to make sure you code is clean and runs in it’s own scope.

First, anytime you create a variable in a function that should not be available outside of the function use the “var” identifier to initialize the variable and restrict it to the local scope.

var item_count = 20;

Second, I would recommend putting much of your code into JavaScript prototype objects. The prototype method of creating objects is JavaScript’s way of creating a class-like object (although prototypes are quite different from actual classes, read up on JavaScript prototypes for more info). In short they allow you to create a group of related function that can share assets between prototype functions (method equivalents).

// start by creating your initial prototype function, like a constructor
function Calendar(month, year)
    this.month = month;
    this.year = year;
    date = new Date();
    this.current_year = date.getYear();

// create functions that inherit characteristics of the prototype Calendar
Calendar.prototype.display_Month = function()
    var max_days_in_month = 31;
    ... continue body of function

// reference variables from the prototype using "this"
Calendar.prototype.display_Week = function()
    if(this.year < this.current_year)
        // return error of some kind
    ... continue body of function

Later in your code you can instantiate one or more of the prototype classes, each having their own scope and assets. This will keep them from conflicting with any other code.

cal_1 = new Calendar(5, 2005);

cal_2 = new Calendar(3, 2006);

If you abstract your code well enough, using the power of JavaScript in this manner allows you to create very reusable code that can be used in any application with any combination of JavaScript without problems. There are other things you can do to abstract your JavaScript and make it more functional, but these examples serve the purpose of resolving scope issues and get you on the road to cleaner, reusable code.

Development, JavaScript , , , , , ,

Sponsored Links

agile ajax black hat Cake PHP centering clifford stoll css cuckoo's egg energy energy drinks espionage flash Flex hacker jquery modular MVC objects optimization performance PHP script timer smarty smarty templates stylesheet up-time uptime variable scope web 2.0 Zend Framework