API Call Limiting
I've gotten a few questions from various clients this past week about API rate limiting. They are under the impression something has changed, or something is broken. I don't believe either is true, but don't feel confident enough in my API knowledge to explain to them exactly what's going on.
I know some other CSMs are fielding these questions as well and so I wanted to start a thread to track the questions/issues that have been reported around rate limiting to hopefully help us answer these questions without having to create support tickets.
Comments
-
Wave:
https://community.waveapps.com/api/v2/discussions?page=1&limit=2500
This call has been used since they implemented the API pipeline with no issues.
As of last week they are getting the following error:
Requesting: discussions | Page: 1
Unable to output data for: discussions. Error: b'{\n "message": "Validation Failed",\n "status": 422,\n "errors": [\n {\n "field": "statusID",\n "code": "invalid",\n "status": 422,\n "message": "statusID is not a valid integer."\n },\n {\n "field": "name",\n "code": "invalid",\n "path": "status",\n "status": 422,\n "message": "status.name is not a valid string."\n },\n {\n "field": "state",\n "code": "missingField",\n "path": "status",\n "status": 422,\n "message": "status.state is required."\n }\n ]\n}'
It appears that the only limit that will now work for this call is "100"
My question is - why would that change?
0 -
Scope:
Using the API with Azure search; running an index of the Community site which recently failed and the API appears to have been changed.
- Looking at the package, I am getting error for the URL https://community.scope.org.uk/api/v2/discussions?followed=false&pinOrder=first&page=1&limit=9999:
- HTTP Exception: The remote server returned an error: (500) Internal Server Error.
I spoke with @Ryan quickly and was able to get back to the client with :
Taking a look at this example - you have your limit set to 9999 - which should not validate.
You should be getting an error that tell you what the cap should be. I'm able to set the limit to about 300 without issue, after that performance starts to diminish
I don't believe this is linked to any change in the code on our side- I believe you may be running into this issue now due to community growth - you're now pulling much more content with that call than you were previously.
The client then let me know that the API call was originally set at 9999 ; however ,the call was changed to a limit of 8000.
They swear the change was not made by devs on their side and what to find out how, why and when this change happened because it caused disruptions on their side.
My question here is - I don't believe that we would be able to make such a change to their API call, and I don't quite understand why the change to a limit of 8000 would cause disruptions.
0 -
I don't think the limit is their issue. I am assuming they have some bad data on one of their posts that is causing it to fail validation during output. When they lower the limit to something sane, the "bad" post isn't coming over on the first "page" of results.
0 -
Couple of notes on rate limiting for CSMS here:
Some notes on counteracting that Tim gave me a while ago for Dentistry in there, but some general notes on the public docs would go a long way, IMHO
0 -
These guys should not be using a limit that high. Input validation should be failing for anything over 100, but it isn't. When we fix it, and we will fix it, their API calls are going to break again. They need to tone it down and start using sane "pages" of results, instead of trying to get everything on the site in one request.
Why did it suddenly "break" for them? They're eating up a ton of resources by exploiting this bug. They're trying to get 10,000 discussions in one single request. I am assuming their site grew to the point where trying to pull all those discussions was more than the server could (or wanted to) handle.
0 -
Just to be clear: passing a "limit" parameter as part of an API request and "rate limiting" are two very different things.
Limiting an API request (using the "limit") parameter is a way of telling Vanilla you want a maximum of X records to be returned in the response.
Rate limiting takes place on our infrastructure, outside of Vanilla (for the most part). It's a way of stopping a user, malicious or otherwise, from making too many requests to our servers.
0 -
Thank you so much, Ryan! Do we have examples of "bad data" that might cause this to fail?
0 -
They had an idea with no status. I fixed it!
0