20

So... we've had a history of bugs with the autocard feature linking to Gatherer. Whatever way we try to link to it, there's always new and different arbitrary cards that break for some reason. Today in chat we discovered an entirely new bug scenario with ordinary card names not working right, which sucks.

Up to this point we've been working on the assumption that Gatherer must be externally accessible as a search engine as long as we just use it right and find the right way to feed it card names. But at this point, I don't think they do try to make it externally accessible.

The bug was: we found out that when a card name is two or more words, and there's another card name that also contains those words, the search fails completely and returns no results. This impacts Waxing Moon (which exists) because there's another card called Cult of the Waxing Moon. This also impacts famous cards like Phage the Untouchable (who also exists) because there are vanguard avatar "cards" for them in Gatherer, like this one for Phage. (But curiously not Etched Champion, which also has an avatar.) When a card is just one word, like Recall or Chronatog, it returns a search page for all cards including that name, without an error.

This stuff really isn't our fault at this point. To date we haven't found a single reliable formula that, for all cards, takes the card name and gives us an exact link to just that card and not a search, and we've gone looking several times. There is no documentation on how to do this correctly nor does Gatherer integrate with years-old modern browser search features, both of which I think they'd provide if they intended to make this system externally accessible.

Then there's also Gatherer's language issue

We discovered recently that Gatherer search fails for anyone from non-English regions. If German is your computer's primary language, Gatherer configures itself to operate in German for you. If you click on an autocard link on Boardgames.SE (because you're also comfortable using an English site), our autocard links tell Gatherer to search for that card name — but it searches for it in the list of German card names. Of course it doesn't find the English card name in the German card list, so the search fails and tells you the card doesn't exist.

This has lead people to edit posts to use straight-up links to the English versions instead of using the [mtg:autocard] syntax. People often won't even know why Gatherer isn't working for them, because they never configured Gatherer themselves. There's no way we know to tell Gatherer to search in the English card list.

gatherer failing a search for grizzly bears
Depicted: the results of clicking an autocard link for Grizzly Bears whilst the site is configured to Japanese.


Let's switch to a different MTG search: Scryfall

(Full disclosure: I'm a Scryfall patron on Patreon and I talk in their Discord channel with other patrons and the team. I have no other affiliation with them and don't stand to benefit in any material way from recommending them. They didn't ask, pressure, or incentivise me to do this in any way, shape, or form. I have checked with them about BG.SE doing this, and linked this page to them.)

Scryfall started up just over a year ago and has become a very popular card search engine in that time. It's wonderful to use and, most importantly for us, has external accessibility as a major development goal. (To the extent they also have a well-developed public API.) Its database is kept up to date with correct details and comprehensive card imagery, and they routinely stay up to date with card previews within a day or two.

... So basically it has everything we'd want. It also has a reliable formula for us to use for searches. And it fixes that language issue by being English-only right now; they have an internationalisation project in their distant future.

Stack Exchange devs: Requesting we use a third-party resource seems weird, so I feel this merits some explanation. M:TG players use a lot of third-party services to augment our involvement in the game. Gatherer plays a role as the single source of truth for Magic card details (including errata, etc) but it is not the go-to search engine. Up to last year the go-to search engine was Magiccards.info, but then Scryfall came out as an effort to modernize MTG searching & deliver things Magiccards.info wasn't doing, and so far they're succeeding.

The search formula

https://scryfall.com/search?q=!"CARDNAME"&utm_source=stackexchange

Scryfall's developers confirmed this one as OK. They requested we include the &utm_source=stackexchange particle for better tracking. In this formula we replace CARDNAME with the card's name: https://scryfall.com/search?q=!"Grizzly Bears"&utm_source=stackexchange

Links below this point all use this formula, although do not contain the &utm_source=stackexchange particle.

One text transformation: strip out double quotes from card names

The only actual transformation we need to apply (beyond normal URI character encoding) to the card name is to strip out quotation marks in the card name (") because they'd mess up that formula. This still works; I'll mention why shortly. Apostrophes (') are fine and don't need to be removed. There's only four cards in all of Magic that use quotation marks in their name, so that means:

This works because Scryfall actually ignores punctuation completely. Even https://scryfall.com/search?q=!"rumorsofmydeath" works just fine. It does this because the developers don't want people to have to remember if Vitu-Ghazi, the City-Tree has one or both of those hyphens in it or not, or if Saskia the Unyielding has a comma or not (like Nissa, Vital Force does). It's a very forgiving mechanism, but still exact on the actual words you use.

Some test cases using this formula

The functional same-name reprints from Unstable result in a search page of all variants, which to my mind is a good thing:

When I say this is a good thing, it's strictly better than what Gatherer does, which is to just return one variant (Very Cryptic Command, Everythingamajig) without hinting that the other functional variants exist.

The nonfunctional variants just return one result since they're handled the same as reprints: Extremely Slow Zombie

Every other card yet mentioned in MTG Autocard bug reports not already mentioned above

(All of these work.)

Some of the above cards have diacritics, the "Æ" grapheme, or punctuation that's unusual for an English writer. Note that I can also write those card names without remembering those details or knowing how to enter those unusual characters with my keyboard:

Details for MTG players

Scryfall has the following plan:

  • Index all printed and online cards accurately, including extras, joke cards, tokens, etc, all with high-res scans
  • Index official spoilers
  • I checked their general policy on leaks:

    Scryfall doesn't post leaked/stolen cards until they're officially spoiled. However, sometimes WotC does "guerilla marketing" where someone opens a pack on a table or in "my attic" etc and we have to guess if it's really an official spoiler or not and we may guess wrong. We also don't count stuff like early release promos or release marketing materials as "leaks" because those get shipped to stores and are almost always posted immediately.

  • 2
    When you say they stay up to date with card previews, do you mean they have officially previewed cards before they're in Gatherer? Do they pick up errata similarly quickly? – Cascabel Dec 15 '17 at 2:28
  • 1
    Also, thanks for figuring this out! Whatever the answer to my question, it seems like it's probably an improvement, and the main issue is just whether SE has time to deal with it. Hopefully at least this API will be much more stable and it'll be a one-time investment without a need for future patches. – Cascabel Dec 15 '17 at 3:13
  • @Jefromi During preview season, they added cards as they were previewed around social media. E.g. here's the Spoiler Date view for Unstable and the Spoiler Date view for Ixalan. They haven't had their own preview card, but they've archived those that showed up. IIRC they also archive leaked cards, such as the sheet of foil promos that was leaked shortly before Ixalan started preview season. – doppelgreener Dec 15 '17 at 9:42
  • 1
    @Jefromi It's my hope as well that it's a one-time switch, without needing to hassle Stack Exchange devs to find ways to fix our interface with gatherer any longer. :) – doppelgreener Dec 15 '17 at 11:09
  • There has been several posts with evidence that the issues we have with gatherer are because the links are not being created correctly and if that was fixed we would not have the issues. – Joe W Dec 15 '17 at 15:04
  • Errata quickly too? I'm not exactly wild about them including leaked cards (and us linking to them) - more visibility/incentive for leaks, and easier for users here to accidentally see leaks they didn't want to. But I suppose it's probably worth it for the reliability in normal cases. – Cascabel Dec 18 '17 at 19:11
  • 1
    @Jefromi We discussed leaked cards on meta here, the current stance being that despite moral issues we pragmatically need to also be free to acknowledge that they happened and examine them because of their influence on the competitive landscape — we don't avoid or ignore leaked cards. If we maintain that policy then a search engine that indexes leaked cards seems reasonable to me. To my knowledge they have integrated errata rapidly (in the order of days, a week at most). – doppelgreener Dec 18 '17 at 19:50
  • @doppelgreener Part of that was simply restating a personal view. However, the conclusion did include treating leaks as true spoilers (with markup, avoiding crucial details in titles, etc) in order to help people who don't want to see them. Adding the ability to easily link to full information for leaked cards tips that balance slightly. Yes, the links can probably also be spoiler-guarded, but it does make it slightly more likely that people will end up seeing unwanted spoilers. Again, I said it's probably worth it anyway, but I do think it's less than ideal. – Cascabel Dec 18 '17 at 19:58
  • @Jefromi I haven't seen that actually happening previously -- nobody's referenced spoiler-season or leaked cards in any question I've seen yet, where that question wasn't specifically about spoiler-season or leaked cards. People could do that even without autocard by just directly linking to Scryfall or Mythicspoiler. (Maybe people could run searches in Scryfall and come across spoiler cards? But lesson learned, during spoiler season searching finds spoilers.) OTOH I want our autocard feature working, and Scryfall is the only site that stays rapidly relatively up to date at the moment. – doppelgreener Dec 18 '17 at 21:15
  • 1
    Yup, agreed, one is a bigger issue than the other. I do think I saw a question or two about leaked cards at some point - maybe some of the BFZ/OGW ones? - so I didn't think this was entirely hypothetical. – Cascabel Dec 18 '17 at 21:21
  • 1
    @Jefromi Asking questions about leaked cards is definitely not hypothetical and has happened, but people talking about leaked cards in answers to questions not about them is not something I've seen. (Unless that's exactly what you're referring to?) – doppelgreener Dec 18 '17 at 21:30
  • @doppelgreener I wasn't making a distinction between answers and questions. I'm just saying that it makes it a very small tick easier for people to accidentally see leaks. – Cascabel Dec 18 '17 at 21:33
  • 1
    @Jefromi I was doubting my memory so I double checked with them about how they handle leaks: they don't. Added a bit to the end of this question. – doppelgreener Jan 3 '18 at 22:50
  • I'm opposed to switching engines. Gatherer is the official source for card text. Only Gatherer is guaranteed to always have the current wording (and the occasionally helpful rulings) for everything. I know that I would always double-check card text, which means more work for no gain. – Hackworth May 27 '18 at 7:30
  • @Hackworth FWIW I think the current plan is to change our gatherer URL to one that works for our test cases. – doppelgreener May 28 '18 at 10:12
13

Yes, please lets do this!

The inconsistency of the current autocard can actually end up taking more time than it saves. When it works it is fine, but when it doesn't you need to head out and find the cards then link them manually anyways. Currently I have taken to double checking my autocard links after I post, if they are broken I have to fix them with an edit (which bumps them on the FP unnecessarily).

I am sure that swapping from Gatherer to Scryfall will be beneficial for them, but personally I don't mind as I think there is a lot to gain for us as well. I think we will have the benefit of being a large driver of site traffic to them, which will probably incentivize them to add features/fix bugs for us quickly in the future.

  • 1
    I agree Scryfall may materially benefit from this arrangement, and that so do we from having a reliable search. Currently they're working to cover their hosting costs via Patreon and the traffic increase may both increase costs and help them cover those costs. – doppelgreener Dec 15 '17 at 17:44
3

FWIW, as of version 1.50 (just released today, January 23, 2018), my SOUP user script now implements this feature (extending and replacing the Gatherer-based reimplementation of [mtg:...] card links included in earlier versions of SOUP).

The fix applies on all MtG card links using the [mtg:...] card link syntax (also in edit preview!), replacing the Gatherer URLs generated by SE with links to Scryfall as suggested above (yes, with &utm_source=stackexchange included). It does not currently try to modify any manually created links to Gatherer or to any other site.

Currently, the links open in a separate browser window, like the current SE card links do. (You can always force them to open in a new tab instead by middle-clicking them.) Given that this version of SOUP also adds support for hover popups for MtG card links, it could be argued that normal links might be more useful for people who actually want to click through and see more than just the card image. Feel free to express you support or opposition to this (or any possible third options) in comments below.

-3

No, I think we need to look at the current system to make sure we are using it properly and sending in search terms to the site in the expected format to return correct results.

Have we reached out to anyone working with the gathering site about the language/card name issue? This could be an issue that they just are not aware about and could work on getting a fix in place.

When testing links using the search string that they suggest on their help page it does return results for these cards even though you still have to pick which card to view which is not ideal.

http://gatherer.wizards.com/pages/search/default.aspx?name=+[%22Waxing%20Moon%22]

I do remember there was a thread with a potential fix for the language issue but I am not fully sure on that

Why are the card links broken?

  • 3
    We've already done that several times though. Here, here, here, here and here all represent different efforts to get our card search working with Gatherer. Each iteration works for some cards but breaks for other cards. The devs implemented one of these or a slight variation; I don't recall; and it has breakages. – doppelgreener Dec 15 '17 at 15:46
  • 1
    Like I noted in the top of the post: we've been working under the assumption that surely there must be a reliable way to get all name queries always working with Gatherer, if only we use it correctly. I don't think that assumption is actually correct. There is no evidence Gatherer is built to be an externally accessible search engine, and there's evidence (like lack of documentation on how to do this) that this is not the case. (In fact we're the only site that I've ever come across that directly reaches out to Gatherer for card lookup.) I'm not in favour of "try to figure it out more". – doppelgreener Dec 15 '17 at 15:49
  • Efforts to fix this problem have already occurred. The autocard links got updated about a year ago based on our suggestions. I believe changing to a new search engine will resolve our autocard lookup issues. – doppelgreener Dec 15 '17 at 15:50
  • Please do, then, since we haven't yet found a solution to make all card lookups work in Gatherer without error. (And there's still the language issue, which is causing an issue for Russian, Italian, etc individuals here in Europe.) I'm proposing "let's dump Gatherer and switch to another search" as a solution to our autocard on the basis we haven't been able to get a reliable Gatherer search working yet. If we can get a reliable Gatherer search query, we don't need to switch. If you have an input formula, we can check it against the test battery we've built up. – doppelgreener Dec 15 '17 at 15:51
  • 1
    That said, I might also be a fan of switching on the basis of Scryfall solving the language issue, which results in people skipping or editing away autocard links altogether as well as opening a couple of recent Autocard bug reports here on meta, whether they understand what's going on behind autocard failing them or not. But that would be less of a "fix the stuff that just isn't working" thing. – doppelgreener Dec 15 '17 at 16:01
  • @doppelgreener I apologize apparently my memory was bad and the best I can do from looking at the gatherer documentation is to return multiple results for cards that share words. Though I still think it might be a good idea to reach out to the people running gatherer to see if they have a solution for this issue. – Joe W Dec 15 '17 at 16:31
  • 2
    Thanks for checking. I think we used to have that, but it also meant things like Sun Titan gave us a search page (because of Sundering Titan) and IIRC cards with apostrophes broke (Pharika's Mender, Ranger's Guile) prompting people to write [mtg:Pharika Mender] instead. – doppelgreener Dec 15 '17 at 17:25
  • 1
    I see that URL doesn't break for Pharika's Mender, which is good. It gives us search pages for e.g. Chronatog or Alive or Alive // Well which isn't ideal though, and it breaks on R&D's Secret Lair. – doppelgreener Dec 15 '17 at 17:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .