Technical update over Link Previews 2.0.
Feel free to scroll away if not interested in the technicalities of this
It turned out to be a bit more complicated given 2 particular websites.
In the final solution, we are crawling websites for their OpenGraph tags, which contain title, description and image of the website to display on the preview, optimising the images, using some reliable fallbacks and this works for the majority of websites, except.... #YouTube
, which as you might have noted, are probably some of the most shared links .
In those cases, its forbidden by their terms of service to do this and they're pretty good detecting automated requests, even if we have separate workers in different geographic regions, rotating workers, together with user-agents.
The official way is submitting a request to use their official API, explaining what we're doing with it and using it to "preview" Youtube Videos and Twitter Tweets.
We have gotten a Youtube API key and link previews work consistently and nicely. Bonus points is that we also get to retrieve some extra information about the video being previewed such as duration, channel name, tags, view counts, etc.
Twitter still doesn't approve our developer usage for this but in the meanwhile, we're already deploying the service.
We have also built the service in a way that in the future we could easily extend it and provide more information about the links previewed such as checking them against malware and phishing databases and checking for the privacy level of the website (based on their SSL configuration, trackers contained and reputation scores from multiple sources such as #DuckDuckGo
and Terms of Service; Didn’t Read).