How Google perfected the web

As the 14th season of Bravo’s Real Housewives of New York City came to a close this fall, I found myself on Reddit, reading rumors about the marriage and divorce timeline of one of the show’s stars. Redditors wanted more clues about a fishy relationship history to see if they could uncover a cheating scandal.

Were divorce papers public record in New York? I wondered. I did a quick Google search to find out.

The search results page was filled with my question’s exact words, repeated across site after site — websites for law firms, posts on forums, ads for creepy lookup tools — but the answer to my actual question was harder to find. At the top of the results page on my phone, Google offered two featured snippets of information quoting different websites. The first one: “Divorce records are not public in New York due to the sensitive nature of many divorce proceedings.” The second: “Due to the state’s underlying legislation regarding family law cases, each divorce is a matter of public record.”

Google bolded both snippets, but it wasn’t clear to me how they squared. I clicked on both.

The two law firm websites were part of an ecosystem I didn’t know existed until I accidentally went looking for it. Law firms across different fields — family law, personal injury, employment lawyers — have blogs full of keyword-addled articles being churned out at a surprisingly fast clip. The goal for firms is simple: be the top result to pop up on Google when someone is looking for legal help. The searcher might just end up hiring them.

Many of these blog posts are written by people like E., a self-employed content writer who juggles law firm clients that want Google-friendly content. E. does not have a legal background; they’re just a competent writer who can turn in clean copy. They trawl health department records, looking for nursing homes that get citations for neglect or other infractions. Then E. writes a blog post about it for a firm, making sure to include the name of the offender and the wrongdoing — keywords for which concerned patients or families will likely be searching. (E. requested anonymity so as to not jeopardize their employment.)

“My bosses, they all don’t want anyone else to know that they use me or that we have the specific process that we have,” E. says. Their name is nowhere to be found, but their writing is often the first thing a searcher will see. The pages were made to be found by people like me.

Google controls around 90 percent of the search market, by some measures, so it’s too valuable a referral source to just leave up to luck. Search engine optimization — or SEO, the practice of tweaking content and websites to get Google to boost your visibility — is everywhere, including on the page you’re reading now. And once you see it or SEO-ify your own work, like E. has, it’s impossible not to notice.

Google’s outsized influence on how we find things has been 25 years in the making, and the people running businesses online have tried countless methods of getting Google to surface their content. Some business owners use generative AI to make Google-optimized blog posts so they can turn around and sell tchotchkes; brick-and-mortar businesses are picking funny names like “Thai Food Near Me” to try to game Google’s local search algorithm. An entire SEO industry has sprung up, dedicated to trying to understand (or outsmart) Google Search. 

The relentless optimizing of pages, words, paragraphs, photos, and hundreds of other variables has led to a wasteland of capital-C Content that is competing for increasingly dwindling Google Search real estate as generative AI rears its head. You’ve seen it before: the awkward subheadings and text that repeats the same phrases a dozen times, the articles that say nothing but which are sprayed with links that in turn direct you to other meaningless pages. Much of the information we find on the web — and much of what’s produced for the web in the first place — is designed to get Google’s attention.

We often hear about the latest engagement hacks on other platforms like Instagram, TikTok, or X, formerly known as Twitter. But Google is consequential above all of these, acting essentially as the referee of the web. Yet deep knowledge of how its systems work is largely limited to industry publications and marketing firms — as users, we don’t get an explanation of why sites suddenly look different or how Google ranks one website above another. It just happens.

Bit by bit, the internet has been remade in Google’s image. And it’s humans — not machines — who have to deal with the consequences.

1. Site performance and accessibility

There’s an inherent contradiction in what Google promises is the best way to succeed on Search. Publicly, Google representatives like search liaison Danny Sullivan give a simple, almost quaint answer to business owners who want help: you just need to make great content for people, not Google’s robots.

At the same time, Google’s “SEO Starter Guide” is nearly 9,000 words long with dozens of links to additional material. There are several SEO industry publications, plus an untold number of scrappy blogs, marketing firms, and self-proclaimed SEO gurus promising to demystify Google’s black box algorithm. Small business owners must either learn how to do SEO or hire someone — even multiple people or special firms — to do it for them. It’s expensive, time consuming, and often confusing work, and failure to learn the ropes could mean trouble if your traffic begins to tank unexpectedly. Google executives like Sullivan often respond to the folk wisdom of the SEO industry with a six-word incantation meant to absolve them of the industry’s worst practices: that’s not what the guidelines say. It can feel like the guidelines are there to protect Google’s reputation, not actually help anyone get search traffic.

Optimizing pages for Google isn’t inherently a bad thing. Google uses its influence over the web to push for objectively good results, like fast-loading sites and accessibility features like alt text on images, which can help audiences understand what’s on a page if an image doesn’t load or if readers use assistive technology like screen readers. Google’s Core Web Vitals metric pushes down sites with certain kinds of intrusive ads or which have slow-loading ads that cause content on the page to shift around.

 “[Google’s changes] did sort of homogenize the design of the internet.”

Perhaps Google’s most benevolent push has been toward a fast, mobile-first web that has forced small and large publishers alike to overhaul their publishing platforms. But even that effort has come with collateral damage — see the entire news industry reluctantly embracing Google’s AMP format — or in the case of smaller blogs, a flattening and whitewashing of web design across the board. 

Valerie Stimac Bailey, a professional blogger of a decade, remembers in 2021 when Google began using a new metric to rank sites, called “page experience,” that emphasized giving readers a “delightful” web to browse. Passing Google’s Core Web Vitals tests became all the more important — Google would look at load times, interactivity, and whether visual elements would move around unexpectedly. 

Bloggers like Stimac Bailey, along with an untold number of other site operators and web companies, saw the writing on the wall: Google might not like your old site, with its giant logos and custom fonts, or the ads that cause text to jump around. Companies like Mediavine, a popular ad-management company, released web design frameworks optimized for this new Google metric and Stimac Bailey, like many others, switched and redesigned her site. But she found the new theme “sterile,” she tells me, and it lacked customization options. It didn’t feel like part of her brand.

“I get that that probably was the impetus for a lot of people with really old, slow themes that were not handling mobile well to move to something that was faster for the world of the mobile-first indexing and internet,” Stimac Bailey says. “That was a good impact… but simultaneously, it did sort of homogenize the design of the internet.”

Stimac Bailey, who in the past published up to 11 blogs at a time, has experimented with different website themes. All eight of her current sites look nearly identical — her Alaska travel blog Valerie & Valise looks the same as Site School, a blog where she shares data-heavy analyses of how her portfolio of websites is performing. 

“People spent a lot of money, and a lot of time, and a lot of heartache and stress and psychology redesigning websites,” Stimac Bailey says.

Taking Google’s advice on creating good, fast, accessible websites sounds nice in theory; why not do what the search engine prefers and help your readers in the process? Creators I spoke to acknowledged that changes sometimes benefit Google and readers alike. But the line between what’s good for the search algorithm and what’s good for audiences has become blurry over time, and in some cases, the two are treated essentially as the same thing.

2. Page design and structure of articles

The small, behind-the-scenes changes site operators deployed over the years have made browsing the web — especially on mobile — more frictionless and enjoyable. But Google’s preferences and systems don’t just guide how sites run: Search has also influenced how information looks and how audiences experience the internet. The project of optimizing your digital existence for Google doesn’t stop at page design. The content has to conform, too.

Take, for example, the question-based subheadings that are rampant on pages ranging from personal finance explainers to travel tips to annual event reminders. Sections like “When should I make IRA contributions?” or “What states are getting rid of Daylight Savings time?” will cascade down a page, presumably to help a reader scan for information. But subheadings are also a piece of information Google uses to understand what a page is about and to rank it in Search. Historically, subheadings have been an easy, fast way to juice content for maximum visibility.

Some bloggers and outlets scrape the “People Also Ask” panel on search results pages for ideas: the Google-curated section spits out strangely worded or oddly specific questions like, “What is the healthiest vegetable 2023?” and “What two vegetables can be eaten raw?”

Sean Bromilow, a food writer based in Canada, has reformatted his blog posts in hopes that Google will pick up his content for placement in these fields. On a page for cucamelons, he added an FAQ section featuring questions like, “How do you eat cucamelons?” and “Are cucamelons a GMO?”

“I did that in direct response to Google’s [People Also Ask questions] that they introduced,” he says.

 Some creators scrape the “People Also Ask” panel for story ideas

A Q&A format might often be the most effective way to write a story or share information — I’ve done stories in this format, too. But other times, question-based subheadings are harder to read, repeating the same phrases without adding anything substantial. Browse this article about gua sha, a massaging technique with roots in traditional Chinese medicine, and you’ll find headings including, “What is a gua sha,” “What are the benefits of a gua sha,” “How to find your gua sha,” and “How to use your gua sha.” 

A table of contents, too, has become a common sight, appearing at the top of articles. On a post about animals to look out for in Alaska, for example, Stimac Bailey has 10 sections in the table of contents, each linking to the corresponding part of the blog post. Having a linked table of contents allows readers to skip to the part they most want to read, like if someone is strictly interested in seeing caribou.

But the table of contents sections also work as jump links on Google Search that appear below the headline and other metadata. Stimac Bailey gets a reasonable amount of traffic to her Pacific Coast Highway guide, not from searchers clicking the title but through people clicking on one of the jump links below. Some SEO strategists even debate whether bloggers should leave their table of contents expanded or collapsed for maximum SEO juice. Stimac Bailey keeps hers collapsed but recently heard from a person selling SEO services that your table of contents should be auto-expanded.

“At a certain point, I don’t care if it costs me time on site or it costs me ad views or costs me bounce rate or whatever it might be,” she says. “I like my site to look the way I want it to look, so that’s what I’m going to do.”

But many websites just do what they think Google wants or what’s being recommended by SEO experts, even if there’s no guarantee it will work. Google is both overbearing with manuals and withholding of clear answers. Give too much away, and everyone could game the system. In that void, creators and website operators throw things at the wall to see what sticks. And once they start designing their page for Google, it’s easy for their content to be fashioned for Google, too.

3. Keyword research and what content is made

For publishers handcuffed to Google Search traffic, there’s often no reason to produce content if people aren’t searching for it. So marketers, writers, and bloggers use a suite of keyword research tools to assess whether there’s enough interest to write the article or make the video in the first place. The result is that publishers end up producing a mountain of material, with Google keywords essentially acting like the assigning editor.

When Stimac Bailey writes for her London travel blog, for example, she strategically picks topics that the site will be able to rank highly for — keywords and topics that are too competitive get put on the backburner.

“[My writers and I] work on picking topics together, but we need them to be productive because not only am I [monetizing them], I’m paying people for their work, and I’m trying to pay very fairly for that work,” she says. “It’s like, ‘I gotta find these low-competition, high-volume, magic keywords.” For a popular destination like London, those magic keywords don’t really exist.

Catherine Cusick understands this tension well. Cusick worked in media for years — including in SEO — before creating the Self-Employed FAQ in March. The subscription-driven business acts as a help guide for people who are new to self-employment or who simply have a specific question they can’t get an answer to elsewhere. 

Most of Cusick’s answers to queries like “Do I need an accountant?” or “What are my healthcare options?” are behind a paywall, so she curates a small number of unlocked articles meant to give prospective customers a sampling of what she offers. These are what Cusick calls “SEO plays.”

For these articles, she is only targeting long-tail keywords — lengthier search terms that are often more specific and, as a result, have fewer people searching for them and are less competitive.

“The keyword search term that I am going for is, ‘How to pay yourself from a single member LLC.’ My game is entirely long-tail keywords,” she says. “I’m not even competing with ‘How to pay myself LLC.’ Like, that’s too high of a term for me, let alone something like ‘LLC.’”

Cusick wrestles with the disconnect between who her business is for — scared, uncertain people trying to make a living — and the SEO requirements she needs to fulfill. Time strategizing and reading technical manuals can feel like time “stolen” from making in-person connections and writing paywalled articles meant to help people through self-employment.

“I will need to have a different page for humans, and then another page that’s more of a directory that points humans who’ve arrived to the directory to other pages that will tell them a story,” she says. “The directory page can be structured in a way that makes search engine crawlers satisfied.” In Cusick’s view, we’re asking one piece of content to do too much: fulfill all the SEO requirements and do the careful, uninterrupted work of getting real answers to a reader.

 I rewrote my prose over and over, but it didn’t seem to satisfy my robot grader

In an emailed statement, Google spokesperson Jennifer Kutz offered a dozen links to public documentation around search, along with generalities about keeping content “helpful” and “relevant.” All points underscored the company’s most common refrain: make content for human audiences. 

“We’ve given longstanding guidance to create content that’s first and foremost helpful, and we work very hard to ensure that our ranking systems reward content designed for people first. Many sites perform well on Search simply by creating this helpful content, without undertaking extensive SEO efforts,” Kutz tells The Verge. “We continuously refine our ranking systems, and where we identify areas we can improve in ranking people-first content, we prioritize them. For more than a year, we’ve had focused efforts to show more content based on first-hand experience in Search, and to reduce content created solely for search engines, and this work continues.”

Kutz did not comment on my questions around specific strategies outlined in this piece, saying that giving granular guidance might make creators “lose sight” of the people-first guidance put forth by Google. Instead, the advice is for website operators to “ask themselves if [a tactic] would be helpful for someone visiting their site.”

But in order to be helpful to readers, website operators need people to visit their site in the first place. Fine-tuning content to match exact search terms is a common strategy that can entice users to click on a page that looks like it will answer their question. That doesn’t guarantee content will be better or even good — and sometimes, how users search can create an echo chamber of errors, oft-repeated misinformation, or poorly researched content.

One instance of errors multiplying sticks out to Bromilow, the food writer. For a while, he says that Google was returning a litany of incorrect information about Ethiopian cardamom, or korarima. Though black cardamom and korarima look similar, their flavors are not. Websites and writers — and by extension, Google results — were confusing the ingredients. At one point, Bromilow says the first picture on Google Images was of the wrong plant. 

“If people are searching the wrong thing because that’s what they’ve been given, how do you return a result to them that explains that they’re incorrect, while also being found by them?” Bromilow says. “You don’t want to reinforce the mistake, right? It’s really weird and complicated.”

That Ethiopian recipes are being translated from Amharic to English also brings a host of problems: how should Bromilow spell the names of dishes? Should he use whatever spelling people are searching for the most? A post on savory pancakes sums it up, in which the Canadian Bromilow explains why he’s opted to omit the “u” in savoury: “The choice, while it breaks my maple-syrup filled heart, is obvious — savory is searched for more often, and using that spelling is more likely to [get] a recipe noticed by the all-powerful and oft-mysterious search engine algorithms.”

To understand what pure SEO-optimized writing looks like, I put my recent story about Google-optimized local businesses through an SEO tool called Semrush that’s reportedly used by 10 million people. 

Among its suggestions: write a longer headline; split a six-sentence paragraph up because it’s “too long”; and replace “too complex” words like “invariably,” “notoriety,” and “modification.” Dozens of sentences were flagged as being confusing (I disagree) — and it really hated em dashes. I rewrote my prose over and over, but it didn’t seem to satisfy my robot grader. I finally chose one thought per sentence, broke up paragraphs, and replaced words with suggested keywords to get rid of the red dots signaling problems. 

The result feels like an AI summary of my story — at any moment, a paragraph could start with “In conclusion…” or “The next thing to consider is…” The nuance, voice, and unexpected twists and turns have been snuffed out. I’m sure some people would prefer this uncomplicated, beat-by-beat version of the story, but it’s gone from being a story written by a real person to a clinical, stiff series of sentences.

Now imagine thousands of website operators all using this same plug-in to rewrite content. No wonder people feel like the answers are increasingly robotic and say nothing.

4.  Building “trust,” Google’s way

If you’re a regular reader of The Verge, you might have noticed some changes to our author bylines in recent months: they’re a lot longer, with more details, name-dropping, and quantifying our professional experiences. You can thank Google for that.

In December 2022, Google updated the metrics it uses to assess the quality of the content it serves up to searchers. Previously, the company looked for expertise, authority, and trust in webpages — now, the company said it would tack on experience to the rating system. In SEO parlance, it’s called E-E-A-T. 

Demonstrating experience by Google’s standards is supposed to show audiences that the person producing the content has participated in the topic matter in some way. If they’re writing about hairdryers, one way to show experience might be a reviewer mentioning how they’ve tested different products themselves. If someone is recommending a restaurant, they could indicate they dined there. The emphasis on experience and trust is in response to issues that, in some ways, are Google’s own creation — on Search, anyone can profit from clicks, including websites spewing disinformation.  

It’s easy to notice these attempts to prove “experience” in the wild: on CNET, author bios have very literal subheadings like “expertise” and “credentials.” Earlier this year, The New York Times began adding enhanced bylines to some stories. A recent investigation into Kanye West’s business deals with Adidas reads, “For this article, Megan Twohey traveled to Portland, Ore., and Los Angeles; interviewed current and former employees of Adidas and of Kanye West; and obtained hundreds of previously undisclosed internal records.” The Times told Nieman Lab the change wasn’t prompted by Google’s policy update, but it’s just the kind of sign of trustworthiness Google likes.

The problem, of course, is that anyone can write up an author bio promising years of expertise — and writing that you’re an expert doesn’t make it true. In November, Sports Illustrated was caught publishing articles attributed to AI-generated authors, complete with specific biographical information and what they like to do in their free time.

 The problem is that anyone can write up a bio promising years of expertise

For reputable news outlets, the challenge of producing work that Google likes and that meets high-quality standards is magnified; you can’t fill articles with SEO-bait keywords without readers noticing. Shelby Blackley, SEO editor at The Athletic, emphasizes that her work’s goal is to get reporters’ stories in front of readers without sacrificing the integrity of the journalism for Google’s algorithms. 

“The philosophy that I’ve always had is the journalism comes first. And regardless of what we try to do from a search perspective, or what Google wants, [that] is always going to be after the journalism,” Blackley tells The Verge.

Blackley co-writes the popular WTF is SEO? newsletter with Jessie Willms, SEO editor at The Guardian. WTF is SEO? is a guide for newsroom SEO editors and audience experts who want to make sure content will reach Google users, but not at the expense of the reporting. Until recently, there were few SEO resources for people in roles like theirs, Willms says.

Showing Google that an author has experience in the topic will look different depending on the type of journalism, says Willms. An investigative team might create an accompanying explainer detailing how they went about requesting records, finding sources, and fact-checking their story; a laptop reviewer, meanwhile, might publish a piece explaining how they assess and test products, so readers know they can trust their reviews.

But even as publications double down to try to prove their experience, and AI-generated synthetic content floods platforms, Google is increasingly trusting and elevating individuals — and sometimes anonymous users. Earlier this summer, some in the SEO industry noticed that Reddit’s visibility on Google was skyrocketing, and Google later revealed it had deployed a “hidden gems” algorithm to surface content from forums or blogs. It’s as if Google saw the discourse about adding “reddit” to the end of search queries and decided to systematize it. Google is also increasingly elevating user-generated content in Search through a “Perspectives” filter that features content from TikTok and other platforms. 

The sleazier SEO strategists have already caught on, waiting and hoping for their chance to exploit whatever Google prioritizes. There are product recommendation articles with titles like “Best Espresso Machine Reddit 2023” and entire websites filled with reviews “according to Reddit” that appear to be fake accounts talking to each other. Some subreddits are overrun with affiliate link spam.

The problem with setting up an ever-scaling system of winners and losers is that someone will always try to cheat and, at least for a bit, they’ll get away with it.

5. Generative AI and the future of Search

Earlier this year, Google unveiled what could become the biggest shift in how people find information online since the advent of Search: results generated using artificial intelligence.

When users opt in to Search Generative Experience (SGE), Google puts AI content front and center, spitting out AI answers and placing them above organic search results. Google has experimented with how to cite sources its AI tool is pulling from, but SGE takes up valuable real estate on the results page, pushing down standard links and potentially killing publishers’ traffic. Why keep scrolling if you can find an answer to your cooking question right at the top of the page?

SEO experts are already thinking about how AI search might change the content they produce. Publishers could double down on creating content in categories that Google won’t include in SGE, for example, like health, finance, or others where the risk of getting it wrong could have negative consequences. Google calls those categories Your Money or Your Life and says SGE, like Search, requires a higher bar for ranking in these areas. Publishers have a way to block Google from training its chatbot, Bard, on their work, but there’s currently no way to opt out of SGE without also disappearing from Search — a death sentence for most outlets.

Some bloggers are also strategizing for the coming AI wave and its potential for destabilizing their business. Zhen Zhou, who runs the cooking blog Greedy Girl Gourmet, recently started a second site focused on traveling in Asia with elderly relatives, believing it’s a niche that is relatively untapped on Google. The hope is that the travel blog and its contents will fare better in the face of generative AI search results than Asian cooking will.

“Maybe after a while the AI will trawl the internet enough to give you just as good recommendations, but at least I think there’s slightly more longevity there,” Zhou says.

 “I have zero desire to satisfy a checklist for generative AI.”

But others, like Cusick, are pulling back. Instead of pivoting to write for whatever role generative AI will play in search, Cusick says she is doubling down on the tactics that work for her, which are often one-to-one, intimate interactions with potential customers. 

“I have zero desire to satisfy a checklist for generative AI or suddenly change tactics in a way that doesn’t align with my values,” she says. These days, outside of Search, Cusick has found success through in-person speaking events or private Discord communities — instead of hitching her wagon to Google and its algorithm, these avenues feel more sustainable.

Though Google has maintained a monopoly on search for years, there are signs that could be changing. The US v. Google antitrust case hinges on Search, its biggest and most ubiquitous product. Other platforms like TikTok are increasingly being used by young people as a search engine, and the video platform has been testing a partnership with Google to insert search queries into TikTok results pages. 

Publishers are already strategizing how their content will need to change as the impending generative AI takeover looms closer. But chasing algorithms has never worked out for the media industry, and increasingly, it looks like this era of search online is waning.

SEO experts like Willms and Blackley are thinking in the long term about how to reliably reach their audiences. Algorithms — and the platforms themselves — ebb and flow; what sticks around is the audience base who comes straight to you.

The habit of prioritizing Google is hard to break — it’s quite impossible to ignore the biggest search company in the world. And though 25 years of living in Google’s world feels eternal, it’s also a drop in the bucket. If Google knows anything, it’s that a young upstart can come along and change everything.

But no matter what happens with Search, there’s already a splintering: a web full of cheap, low-effort content and a whole world of human-first art, entertainment, and information that lives behind paywalls, in private chat rooms, and on websites that are working toward a more sustainable model. As with young people using TikTok for search, or the practice of adding “reddit” to search queries, users are signaling they want a different way to find things and feel no particular loyalty to Google.

It’s all but certain that a new era of content relentlessly optimized for AI search engines is bound to result in the same kinds of problems we have today. And with that realization, there is a chance at another way forward, without the almost religious dependence on Google; searching and creating for a vacillating overlord feels increasingly futile. And when the creators and searchers leave the web for good, Google will have nobody to blame but itself.

The post How Google perfected the web appeared first on AIPressRoom.