We are just testing the new TMDB TVShow scraper for Kodi (that will be default in the next version of Kodi) and coming up against 429 rate limit errors.
More info here: https://forum.kodi.tv/showthread.php?tid=200504&page=11
Testing shows like "The Walking Dead" which have 132 episodes in the series and Kodi is clearly going to hit the API limits.
So my question is, can the rate limits be expanded now TMDB supports TVShows? It seems they are a little restrictive for TVShow API calls. Or is the Kodi scraper just not doing it as efficiently as it should perhaps?
Can't find a movie or TV show? Login to create it.
Want to rate or add this item to a list?
Not a member?
Reply by Travis Bell
on November 21, 2018 at 1:04 PM
Hey @zag2me,
Do you know what calls are being made? Specifically, if individual episodes are being called, or is it a season call?
Will return an
episodes
array containing all of the episodes of the season for example. So they should be able to get away with only calling a single HTTP call for all of the episodes per season. But I am not sure on which calls are being made or how they are structured in the Kodi scraper.Reply by zag
on November 21, 2018 at 2:57 PM
Yeh looks like the current scraper is using the episode lookup:
I'll see if we can get the scraper updated to what you suggest.
Reply by zag
on November 22, 2018 at 4:00 AM
Thanks! the Kodi scraper has now been updated to use the series API call. Works well.
I can't help but think the API rate limit is still very low though. If we scrape in say 100 series with an average of 3 seasons (for example) that's at least 300 requests which is going to be very slowed down if only 40 are allowed each 10 seconds.
Was the API limit designed in a time when TMDB supported TVShows? Does the API rate limit reset when we move onto the next TVShow?
Reply by Travis Bell
on November 23, 2018 at 3:11 PM
The rate limits were added after TV was added, yes. I've never really heard from anyone about this too much. There's lots of ways to do multiple requests in one by using
append_to_response
.There's some basics like so:
But if you know that a season exists, you can even do this (get season 1 and 2 in a single call):
So it kind of all just depends on the specific use case and need.
Reply by zag
on November 25, 2018 at 9:14 AM
Thanks for that, I think it doesn't help the Kodi so much as it would add quite a lot of complexity to the scraper.
See here for more : https://forum.kodi.tv/showthread.php?tid=200504&pid=2794769#pid2794769
I think we also lost multi episode thumbs compared to the TheTVDB scraper, but that's not a massive loss.
Reply by Travis Bell
on November 25, 2018 at 11:23 AM
Perhaps there's a tweak that Kodi could make in the future for a way to view all of the images of a particular episode? This would make a call to the detailed episode method with images (instead of what I would assume it does today, and reads the cached data?) I'm just looking at Karellen's example where they weren't happy with the highest rated image of a particular episode.
Reply by Marek71cz
on November 17, 2020 at 5:10 AM
I have sometimes problem with this scraper, exception raised, but no info about it, another error, see
Reply by Travis Bell
on November 20, 2020 at 11:55 AM
Hi @Marek71cz, looks like that scraper is tripping over some unicode. I would suggest heading over to the Kodi forums and post more details there. I don't maintain the scraper so I am not really able to help much.