Wave needs help with pulling data over the API

Unknown
Unknown
in Help

Hi There,

Wave has been having some issues with our API, both from the call limiting (which was patched in the most recent release) but i don't believe has hit their VIP cluster yet and with "bad data". Which in their case meant ideas without a status

The history of these issues can be found in this task: https://vanillaforums.teamwork.com/#tasks/10795987 as well as within this forum discussion: https://staff.vanillaforums.com/discussion/606/api-call-limiting#latest

While the Issue of "bad data" seems to have been resolved, in my opinion. They still seem to be running into issues with what they are trying to do.

They summarized their issues below:

As discussed today, here you can find a sum up of the issue we currently have:


- the Comments API (https://docs.vanillaforums.com/help/apiv2/swagger/#/Comments/get_comments) allows only to get comments by Discussion Id. Wave currently has more than 100 discussions, a number that grows every month. With the current API system, we need to run first an API call to retrieve all the Discussion ID, save the list and then run a call (or multiple in case of pagination) to the comments API per each Discussion ID, save all the results and then update the Comments table in our database. We would like to have an API option to get all the comments, so without providing a Discussion ID. Pagination does not represent a problem.


From the previous call we had with Vanilla (Charlotte can confirm) we know that other companies have already raised the same issue and they want to provide a solution.


Those are all the information I have. If you need something else before your meeting please let me know.


Thanks for your help, Craig


Separately, we've encountered errors working with the API that have resulted in inconsistent or inaccurate updates. In short: working with the data is absolutely crucial for our operations and until the API is viable, we're limited to workarounds. In this case the recommended workaround - a data export - creates an abundance of extra work. If we have to operate with this workaround the viable solution would mean we could export the tables into individual CSVs (something that with the tool your team uses, Sequel Pro, takes minutes once an .sql export is loaded) and have those added to an S3 bucket. 


In a perfect world, we'd stick with the API: having data that is a month dated and requires manual requests from me (and currently an immense amount of work for our data team) is very limiting. 

After speaking with @Linc about this briefly, he mentioned that this might be an issue due to the fact they had a migration. They migrated to us from Zendesk.

They have been suffering from these issues since about May and would love to get on a dev call, which I've agreed to. But would like our dev team to get acquainted with the issue before getting on a call.

So i'm tagging @Ryan as the foremost authority on API and @patrickdesjardins as our migration dev extraordinaire.

Comments

  • Unknown
    Unknown

    It sounds like they're building a scraper and want to be able to grab all comments without specifying the currently-required criteria (a discussion ID or user ID). This isn't something the API currently allows, but I suppose it could.

    I've created an issue on the core repo (vanilla/vanilla#8090) to at least open it up for discussion.

    In the meantime, it looks like Wave is using Advanced Search, which includes the search endpoint. They can take a look at using it to pull comment data. It won't have all the same fields as the comments endpoint, but maybe it's enough for their purposes. Some sort of scope/filter has to be sent with the search query, but they can use the dateInserted field with a date that predates their content as a workaround. For example: https://community.waveapps.com/api/v2/search?recordTypes=comment&dateInserted=>2000-01-01&limit=100. Please note: this isn't what search was intended for, but if the client needs an immediate solution and is getting ornery, this is an option.

Categories