Switch Magic autocard over to a different search engine, Scryfall

Question

So... we've had a history of bugs with the autocard feature linking to Gatherer. Whatever way we try to link to it, there's always new and different arbitrary cards that break for some reason. Today in chat we discovered an entirely new bug scenario with ordinary card names not working right, which sucks.

Up to this point we've been working on the assumption that Gatherer must be externally accessible as a search engine as long as we just use it right and find the right way to feed it card names. But at this point, I don't think they do try to make it externally accessible.

The bug was: we found out that when a card name is two or more words, and there's another card name that also contains those words, the search fails completely and returns no results. This impacts Waxing Moon (which exists) because there's another card called Cult of the Waxing Moon. This also impacts famous cards like Phage the Untouchable (who also exists) because there are vanguard avatar "cards" for them in Gatherer, like this one for Phage. (But curiously not Etched Champion, which also has an avatar.) When a card is just one word, like Recall or Chronatog, it returns a search page for all cards including that name, without an error.

This stuff really isn't our fault at this point. To date we haven't found a single reliable formula that, for all cards, takes the card name and gives us an exact link to just that card and not a search, and we've gone looking several times. There is no documentation on how to do this correctly nor does Gatherer integrate with years-old modern browser search features, both of which I think they'd provide if they intended to make this system externally accessible.

Then there's also Gatherer's language issue

We discovered recently that Gatherer search fails for anyone from non-English regions. If German is your computer's primary language, Gatherer configures itself to operate in German for you. If you click on an autocard link on Boardgames.SE (because you're also comfortable using an English site), our autocard links tell Gatherer to search for that card name — but it searches for it in the list of German card names. Of course it doesn't find the English card name in the German card list, so the search fails and tells you the card doesn't exist.

This has lead people to edit posts to use straight-up links to the English versions instead of using the [mtg:autocard] syntax. People often won't even know why Gatherer isn't working for them, because they never configured Gatherer themselves. There's no way we know to tell Gatherer to search in the English card list.

^{Depicted: the results of clicking an autocard link for Grizzly Bears whilst the site is configured to Japanese.}

---

Let's switch to a different MTG search: Scryfall

(Full disclosure: I'm a Scryfall patron on Patreon and I talk in their Discord channel with other patrons and the team. I have no other affiliation with them and don't stand to benefit in any material way from recommending them. They didn't ask, pressure, or incentivise me to do this in any way, shape, or form. I have checked with them about BG.SE doing this, and linked this page to them.)

Scryfall started up just over a year ago and has become a very popular card search engine in that time. It's wonderful to use and, most importantly for us, has external accessibility as a major development goal. (To the extent they also have a well-developed public API.) Its database is kept up to date with correct details and comprehensive card imagery, and they routinely stay up to date with card previews within a day or two.

... So basically it has everything we'd want. It also has a reliable formula for us to use for searches. And it fixes that language issue by being English-only right now; they have an internationalisation project in their distant future.

Stack Exchange devs: Requesting we use a third-party resource seems weird, so I feel this merits some explanation. M:TG players use a lot of third-party services to augment our involvement in the game. Gatherer plays a role as the single source of truth for Magic card details (including errata, etc) but it is not the go-to search engine. Up to last year the go-to search engine was Magiccards.info, but then Scryfall came out as an effort to modernize MTG searching & deliver things Magiccards.info wasn't doing, and so far they're succeeding.

The search formula

https://scryfall.com/search?q=!"CARDNAME"&utm_source=stackexchange

Scryfall's developers confirmed this one as OK. They requested we include the &utm_source=stackexchange particle for better tracking. In this formula we replace CARDNAME with the card's name: https://scryfall.com/search?q=!"Grizzly Bears"&utm_source=stackexchange

Links below this point all use this formula, although do not contain the &utm_source=stackexchange particle.

One text transformation: strip out double quotes from card names

The only actual transformation we need to apply (beyond normal URI character encoding) to the card name is to strip out quotation marks in the card name (") because they'd mess up that formula. This still works; I'll mention why shortly. Apostrophes (') are fine and don't need to be removed. There's only four cards in all of Magic that use quotation marks in their name, so that means:

• If I write [mtg:"Ach! Hans, Run!"], strip out the quotes to make it just Ach! Hans, Run!: https://scryfall.com/search?q=!"Ach! Hans, Run!"
• Same for [mtg:Kongming, "Sleeping Dragon"]: turn that into Kongming, Sleeping Dragon: https://scryfall.com/search?q=!"Kongming, Sleeping Dragon"
• [mtg:Pang Tong, "Young Phoenix"] becomes Pang Tong, Young Phoenix: https://scryfall.com/search?q=!"Pang Tong, Young Phoenix"
• [mtg:"Rumors of My Death . . ."] becomes Rumors of My Death . . .: https://scryfall.com/search?q=!"Rumors of My Death . . ."

This works because Scryfall actually ignores punctuation completely. Even https://scryfall.com/search?q=!"rumorsofmydeath" works just fine. It does this because the developers don't want people to have to remember if Vitu-Ghazi, the City-Tree has one or both of those hyphens in it or not, or if Saskia the Unyielding has a comma or not (like Nissa, Vital Force does). It's a very forgiving mechanism, but still exact on the actual words you use.

Some test cases using this formula

• The Longest Card Name Ever Elemental
• Split cards can be searched by either or both halves: Alive, Well, Alive/Well (the traditional double slashes are also fine: Alive//Well)
• Aftermath cards too: Insult, Injury, and single/double/triple slashes are all fine: Insult/Injury, Insult//Injury, Insult///Injury (since they ignore punctuation anyway)
• Cards mentioned above all return their exact correct matches without errors or a search page: Phage the Untouchable, Waxing Moon, Cult of the Waxing Moon, Etched Champion, Chronatog, Recall
• The card simply named X
• Searching for Storm Crow returns both the actual card and the token variant generated by Crow Storm.
• Searching for a token by name returns each unique functional variant of that token: Soldier

The functional same-name reprints from Unstable result in a search page of all variants, which to my mind is a good thing:

• Very Cryptic Command
• Garbage Elemental
• Knight of the Kitchen Sink
• Ineffable Blessing
• Sly Spy
• Everythingamajig

When I say this is a good thing, it's strictly better than what Gatherer does, which is to just return one variant (Very Cryptic Command, Everythingamajig) without hinting that the other functional variants exist.

The nonfunctional variants just return one result since they're handled the same as reprints: Extremely Slow Zombie

Every other card yet mentioned in MTG Autocard bug reports not already mentioned above

(All of these work.)

• Circle of Protection: Art
• Ranger's Guile
• R&D's Secret Lair
• Question Elemental?
• B.F.M. (Big Furry Monster)
• Sphinx's Revelation
• Sun Titan
• Sundering Titan
• Black Lotus
• Erase (Not the Urza's Legacy One)
• Ghazban Ogress
• El-Hajjâj
• Ring of Ma'rûf
• _____
• Framed!
• Sauté
• Who/What/When/Where/Why, accessed by the full name.
• Yet Another Æther Vortex
• The Ultimate Nightmare of Wizards of the Coast® Customer Service
• Chicken à la King
• Junún Efreet
• Pharika's Mender
• Lightning Bolt
• Platinum Angel
• Grafdigger's Cage
• Trial / Error (and Trial and Error)

Some of the above cards have diacritics, the "Æ" grapheme, or punctuation that's unusual for an English writer. Note that I can also write those card names without remembering those details or knowing how to enter those unusual characters with my keyboard:

• El Hajjaj
• Ring of Maruf
• Saute
• Chicken a la King
• Junun Efreet
• Yet Another Aether Vortex

Details for MTG players

Scryfall has the following plan:

• Index all printed and online cards accurately, including extras, joke cards, tokens, etc, all with high-res scans

• Index official spoilers

• I checked their general policy on leaks:

  Scryfall doesn't post leaked/stolen cards until they're officially spoiled. However, sometimes WotC does "guerilla marketing" where someone opens a pack on a table or in "my attic" etc and we have to guess if it's really an official spoiler or not and we may guess wrong. We also don't count stuff like early release promos or release marketing materials as "leaks" because those get shipped to stores and are almost always posted immediately.

Answer 1

Yes, please lets do this!

The inconsistency of the current autocard can actually end up taking more time than it saves. When it works it is fine, but when it doesn't you need to head out and find the cards then link them manually anyways. Currently I have taken to double checking my autocard links after I post, if they are broken I have to fix them with an edit (which bumps them on the FP unnecessarily).

I am sure that swapping from Gatherer to Scryfall will be beneficial for them, but personally I don't mind as I think there is a lot to gain for us as well. I think we will have the benefit of being a large driver of site traffic to them, which will probably incentivize them to add features/fix bugs for us quickly in the future.

Answer 2

FWIW, as of version 1.50 (just released today, January 23, 2018), my SOUP user script now implements this feature (extending and replacing the Gatherer-based reimplementation of [mtg:...] card links included in earlier versions of SOUP).

The fix applies on all MtG card links using the [mtg:...] card link syntax (also in edit preview!), replacing the Gatherer URLs generated by SE with links to Scryfall as suggested above (yes, with &utm_source=stackexchange included). It does not currently try to modify any manually created links to Gatherer or to any other site.

Currently, the links open in a separate browser window, like the current SE card links do. (You can always force them to open in a new tab instead by middle-clicking them.) Given that this version of SOUP also adds support for hover popups for MtG card links, it could be argued that normal links might be more useful for people who actually want to click through and see more than just the card image. Feel free to express you support or opposition to this (or any possible third options) in comments below.

Answer 3

We still should definitely do this

Most of the issues with using gatherer have been solved, new updates to how the autocard works with gatherer have fixed a lot of them, but there is an issue that no one has mentioned here yet - Gatherer has issues with uptime. I have never seen Scryfall down, but I see gatherer down once or twice a month at least (including when I was doing my L1 judge test, made it a lot more difficult because they too use gatherer's hosted card images, I just searched the cards up on scryfall).

I am English speaking and gatherer for me is in English, so I don't know if that part of the original post is still an issue, but I've found Scryfall to be simply more robust in every way, and I know it does not have that language issue.

Answer 4

No, we should stay with Gatherer.

I see the advantages of Scryfall and shortcomings of Gatherer search. However, there is one crucial point in favour of Gatherer that outweigh any inofficial search engine: It is the official source of all MtG card wordings.

108.1. Use the Oracle card reference when determining a card’s wording. A card’s Oracle text can be found using the Gatherer card database at Gatherer.Wizards.com.

This implies that Gatherer wordings are, by definition, always correct and always in sync with Oracle changes. If there was ever any dispute over a rules question that depended on the wording of a card, a Gatherer reference would always win out over a non-Gatherer one, and double-checking cards from a non-Gatherer source with Gatherer would only add extra steps.

It also has official Gatherer rulings. They usually are just re-statements of the relevant rules, but can occasionally help with tricky questions such as this..

Answer 5

No, I think we need to look at the current system to make sure we are using it properly and sending in search terms to the site in the expected format to return correct results.

Have we reached out to anyone working with the gathering site about the language/card name issue? This could be an issue that they just are not aware about and could work on getting a fix in place.

When testing links using the search string that they suggest on their help page it does return results for these cards even though you still have to pick which card to view which is not ideal.

http://gatherer.wizards.com/pages/search/default.aspx?name=+[%22Waxing%20Moon%22]

I do remember there was a thread with a potential fix for the language issue but I am not fully sure on that

Why are the card links broken?