Wikipedia and friends: learning through code

As part of my thinking on the CC board about open, the state of the web, and the state of Wikipedia, I’ve become increasingly interested in how Wikipedia relates to the rest of open knowledge. Is it connected? Is it possible to connect it?

Inspired by that question, and some conversations I had at Wikimania, I have built friendsof.wiki. Friendsof tries to transform linked data into surfaced content from “friends of open”, using a Wikipedia article as the framework to showcase the entire world of open.

The Night Watch (detail) by Rembrandt van Rijn, 1642, Rijksmuseum, public domain. Fetched via the museum’s Linked Art API — one of the friends this post is about.

This is still an experiment, a learning tool, and a WIP. For performance reasons, I don’t think it can become a “real” way of browsing the wiki, and there is so much linked data that usable automated curation and layout is a hard problem! But that’s OK: like I said, a lot of my goal was understanding the state of the open knowledge graph; if this specific implementation is not productizable that’s fine. (That said, if you like it and want to contribute in some way, or have questions, it’s on my github.

So here, some lessons I learned while building it.

#The good news

My post on the conflicting graphs of open told an interesting story: Wikipedia’s web traffic is falling off a cliff, but by many other metrics open knowledge (like arXiv) and open creation (like GitHub) is thriving. There are many partners from both new forms of knowledge (like iNaturalist and GBIF) and traditional institutions (like the Met and the Rijksmuseum). In many ways, open knowledge has never been healthier.

Toying around with this has confirmed the underlying story for me: there’s so much great stuff out there, reliably and algorithmically reachable from open’s flagship. My favorite finds from this, so far: 3d models of the Apollo spacecraft from the Smithsonian; this 1970s remix of American Gothic in support of the Equal Rights Amendment; and the excellent presentation of the Night Watch at the Rijksmuseum.

This does require that Wikipedia embrace that role of open flagship, though. At least in theory, we’re already there; in 2017 we said that:

By 2030, Wikimedia will become the essential infrastructure of the ecosystem of free knowledge, and anyone who shares our vision will be able to join us.

A lot has changed since we wrote that, but I still find that vision quite inspiring. So… how are we doing?

#Our opportunities

I think there’s a lot we can do that would help us build towards the vision laid out in the movement strategy.

#We need to break down some barriers to partnership

Bluntly, our current rules about third-party content and links mean that at best we are going to be a bad partner, because we deliberately force almost all partners to either dissolve into Commons (because we prohibit third-party images in articles) or become a mere line of text. As long as that’s our default position, “join us” really means “abandon your own identity” — it isn’t partnership, it’s absorption.

To be fair, this is not an easy problem to solve. Other sites really do have quality challenges — we don’t want to have bad (or worse, spammy) content. Picking winners and losers would not be fun — if 37 museums have Rodin sculptures, how do we know who to link to? Third-party javascript on the site has both technical and legal challenges — we want our privacy guarantees to really stick. Etc etc. But all of them seem navigable — if we have the will to do it.

#We’re missing opportunities to promote free knowledge in user-friendly ways

By refusing to make any choices at all, and instead presenting an unprioritized scroll of (currently) 572 options, our ISBN page has long been a lesson in user-hostility. José Rizal’s page, as one example, has one ISBN-linked book he wrote. The citation gives you two options: the Google Books link, or the ISBN, styled exactly the same, landing on that 572-link scroll. His bibliography in the same article has a similar user-hostility problem—not linking helpfully to the books themselves. By pulling from Open Library, we can do better, on both counts.

Wikipedia’s bibliography for Rizal (top) names both novels and shows neither cover. friendsof.wiki asks Wikidata for his Open Library author ID instead — getting links directly to his most important books.

This experiment also led me to learn about {{Taxonbar}}. It lists 23 different places to get information about taxons, including GBIF and iNaturalist. This would be fine except those are two of the fastest growing members of the “ecosystem of free knowledge” we said we were going to promote in 2017 — this template really should highlight them, perhaps paralleling what we do with some open-access links.

The same two identifiers, before and after.

Putting together this screenshot made me see several important data and usability regressions between the usual infobox and my demo “infobox” replacement. That’s OK; when I fix them in my demo, they’ll be fixed for every article about a species, not just “Monarch”.

This would have been a lot harder 10-15 years ago: we would have been doing our users a disservice by linking to bad sources just because they were free. But these two projects are now huge, as is Internet Archive’s giant set of old public domain book scans, among others. So the choice is now different, and we should think about how to serve our readers while also supporting our strongest possible allies — especially in this challenging moment for all knowledge.

#Wikidata matching needs love

Magnus and others in the GLAM community have done some great work, but there is much more to do. As just one example, right now from my data spelunking it seems that for small catalogs, sometimes someone falls in love and grinds through it. (If your catalog is under 1k links, 58% are likely to be matched.) Large catalogs — the most impactful, across the widest swathe of knowledge — don’t get finished. (If your catalog is over 1M, 6% are matched.) There are ~145M entries unmatched in mix’n’match, and more databases that have simply never made it in.

This is not a critique of mix’n’match or the people who use it. I think it’s just a natural outcome of this sort of data matching being both hard and sort of boring to do literally millions of. So how do we apply tools, gamification, etc., to fix that?

#Treating live-data-checking as a first-class source

It appears we have challenges about keeping matched data up-to-date. Two examples I hit while building Friendsof: Wikidata’s iNaturalist ids are in some ways a success story, with ~626k matches through Mix’n’match catalogue #238. Our import is still working from a snapshot from 2021, but since then iNaturalist has reorganised fungal taxonomy, and as far as I can tell none of that has been synced back to Wikidata. Similarly, GBIF re-keyed its taxonomy this year. The original property, P846, is now labelled “GBIF-species-ID (before 2026 update)” and has three million uses. Its replacement P14607 has less than a thousand uses in Wikidata, so Taxonbar now supports both.

Neither of these are the end of the world, but they do suggest we need systematic and ongoing automated review of links to partner orgs. One-time imports are not going to be enough if we want to be a good peer node to our partners in open knowledge.

#Tools are our friends

We live in a golden age of text processing, and so filling the gaps left by Mix’n’Match is exactly the sort of boring-at-scale work machines could be doing. But that runs square into Wikidata’s trust issues with bots. Part of this is inevitable and unavoidable, and can only be solved by careful work on the part of bot authors. But it seems to me like we’re leaving low-hanging fruit on the floor by not implementing mandatory property constraints. (We know such checking can improve quality: YAGO imposed schema.org typing on Wikidata, and found 132M type-inconsistent statements. Surely some were false positives, or fixable, but it still suggests a huge opportunity for us to improve our quality.)

#We must grapple with licensing ambiguity

Some of our best partners should be archives and libraries — who rely heavily on fair use and similar exceptions. And some of our most exciting partners are big data repositories, where licensing is an inherently complex and ambiguous endeavor. Linking out to them will point us to things (like the 1970s ERA poster above, and complex taxonomic data) whose license status is obviously educational in nature, but not vetted in the same way Commons is. This is going to be uncomfortable for us; we like to pretend that licensing is a clear binary and that’s… not how things are, or ever could be, for many of our closest ideological allies.

I do not have a good proposal for what that would look like — but it is clear we will need to be flexible in ways we have not been in the past. (And this is one area where our partners really could perhaps come towards us, cleaning up their metadata to help make us more comfortable.)

#What can we do, at the highest level?

The previous points are somewhat nitpicky, operational stuff. Here are some of my bigger takeaways on what we should aspire to change to help us meet the challenges of the current moment.

#We need partner-level equivalents of Assume Good Faith and Be Bold

The primary curation model we offer to our partners is slow manual addition: prove a partner belongs, one template/one link at a time, forever. That’s not unreasonable: we need to have high standards. But we have the opportunity to partner with some of the most esteemed institutions on earth, and some of the fastest-growing free knowledge projects. They deserve our patience — the same patience we show human editors. And they need to see results, not eternal grinding of gears.

In short, I suspect we need partner-level versions of both AGF and BOLD: find good-faith partners, boldly include their content, with linking, by default, then curate actively. That would of course have to include helping them improve their data and licensing quality — but would also have to include us committing to being a good partner when they do that work. The evidence suggests that second part is still a work in progress.

#The best thing we can do for Wikidata is surface it

I didn’t start this experiment to learn about Wikidata’s quality—my focus was going to be on how to build bridges from it to the rest of open. But the act of transforming invisible links into visible content uncovered so much that I can work on.

Mostly, this is great: stuff to do! Improvements to make! But it also made it very clear to me that, like most platforms, the best way to improve the quality of the platform is to build something that really pushes the platform hard—something that is not just visible to the builders, but visible to many users. Wikidata has not historically done that very well (often for perfectly good reasons).

Historically, enwiki’s interpretation of this mixed quality is “therefore we can’t rely on it”, which becomes circular: our biggest human surface, with the most editors, can’t use it; therefore it gets few editors; therefore it doesn’t get improved; therefore our biggest human surface doesn’t use it.

I would take the opposite view: the only way we can improve the quality of enwiki and wikidata, together, is by using the data. The first step to fixing problems is to bring them into the light.

There’s a lot of hard detail here, of course — I’ve got the outlines of a whole post on the many small things I’ve already found. I suspect a team of editors working together to do this embedding could do a lot of good quickly. If you know of such a group, let me know! I’d love to join!

#Recommit ourselves to partnership

It’d be great if we can transition from an island of open to being part of an archipelago — a connected network. I hope this little experiment gives other folks some ideas on how to make that happen. I know it has for me!

#LLM usage

I built friendsof.wiki with Claude Code.