Unindexed — the website built for Google to destroy (2024)

The art-technology project explored the permanence, or lack thereof, of the web

When Matthew Rothenberg created a new website in early February, he let about two dozen people know about it through an unlikely medium: postcards.

The unorthodox method was fitting for an unorthodox website called Unindexed. It was the latest project from Rothenberg, a 35-year-old based in Brooklyn, who has created a portfolio of interactive web installations and performance art projects around technology.

Unindexed is no more. The website was coded to erase itself once Google added it to its search index. It lasted a little over three weeks, disappearing forever on Feb. 24.

Rothenberg has done stints as head of product for Flickr and Bitly but for the last couple of years has focused on consulting and his art-technology side projects. His goal for Unindexed was to create a site where people could post comments safe in the knowledge that no record of those posts would ever exist again. It was also coded to prevent Google from caching it.

It was an experiment exploring the web’s dichotomy: how it at times it can feel permanent, and at other times ephemeral.

“We’ve learned to be very careful and to mediate ourselves in the way we communicate online, and people don’t really feel free,” he said in a phone interview Monday. “I was curious, if I created a space where people could speak without feeling they were on a permanent record, what would happen.”

In some ways, the web is still a fairly young medium, and people are still coming to grips with how permanent or ephemeral information they post there can be. Some things endure online for a very long time, while some services that seem stable can disappear.

In 2009, Yahoo shut down GeoCities, a large online community that let people create simple web pages and hosted other features such as bulletin boards and chat. It felt dated after being online for 15 years, but people reacted strongly when Yahoo scuttled it, and the Internet Archive launched a project to archive its content.

Rothenberg was unsure if Unindexed would live even for a day. If someone posted a link to it on a social network, that could bring it to Google’s attention and the site would destroy itself.

There’s certainly interest in applications that don’t retain records. The popular Snapchat app allows people to send photos that disappear from recipients’ devices. Wickr, a multimedia messaging app, lets messages self-destruct after a certain time.

Unindexed was even more fragile, and open on the web rather than a private messaging service. Its survival depended on a quality that seems rare these days: its users’ self-restraint. The action sought after by so many app developers — that people who discover a service share it with others — could also kill it.

Those who received Rothenberg’s postcards were warned that sharing information about the site online would lead to its demise. But he didn’t advise them against it.

Rothenberg posted a short essay on the site outlining his thoughts. Writing doesn’t come easy to him, he said, and he often agonizes about what he posts online. But in this case, “I didn’t feel the same degree of pressure because I knew it wasn’t permanent. It felt like somewhat of a safer space for me to write,” he said.

The site had a few hundred visitors and was apparently shared discreetly. While the site stayed under Google’s radar for more than three weeks, it was quickly visited by spam bots that posted random sentences.

Unindexed was “completely out there in the dark,” a random domain name among millions promoted only modestly through post cards, “yet there are these bots out there that are searching for somewhere they can shout into,” he said.

He’s unsure how Google eventually latched onto the site, though it was only a matter of time. Unindexed was coded to constantly search for itself on Google to see if it had been spotted it. Its disappearance was something of a relief.

“I feel very much the kind of pressure of maintaining things forever,” he said. “Every time I build a project and put it on the web, it’s kind of expected that people are going to be able to see it in a year or five years. The reality is that software is very, very brittle. It breaks all the time. It needs constant attention. It’s a very needy child.

“The more and more projects that I do, the more and more time I have to spend in the past just maintaining things. For me, there was something a little cathartic about knowing it’s going to be gone.”

Anyone wanting to repeat the experiment can find Unindexed’s code at Github, and Rothenberg’s explanation of the experiment, and other projects, on his website.

Send news tips and comments to jeremy_kirk@idg.com. Follow me on Twitter: @jeremy_kirk