Using Botify to monitor SEO in Vanilla?

Unknown
Unknown
edited February 2020 in Help

I am current in talks with an opportunity for a media company called CarBoatMedia. They own this forum here: http://forum-auto.caradisiac.com/

They initially asked the following:

"We use Botify for monitoring our SEO. Currently,it's taking. logs from AWS to the monitoring. Is it a practice that you are comfortable with?we deposit on a bucket S3 access log from our website every day at 4 AM. Regarding EU GDPR, we must anonymize our IP MD5"

I replied the following:

I am cautiously saying I do not see a problem with this. I've never used Botify and it should really be Botify telling you whether or not a changing platforms could change how this works. Based on the documentation on the Botify site, they state that you don't even need to own the domain to be able to do a basic analysis so it seems like it should still work no problem. I do not see how Vanilla would be involved in sending the information from Botify to AWS. It seems to me that you choose your domain to crawl and Botify sends you the information. This should still work without interference. 

They then attached the following document:

I still don't understand how this part of the equation would involve Vanilla. Am I cazy here or does anyone else understand what they're getting?

I don't know how Botify exactly works but I read on the documentation it seemed to do with crawling a site. Do they need any special access to Vanilla to be able to do so? Part of me wonders if it is a language barrier but I don't seem to be able to give an answer that it will still work.

Any thoughts?

@Rhys @Adrian

Comments

  • Alex Powell
    Alex Powell

    I would like to defer to ops than assume anything.

  • Unknown
    Unknown

    I agree with you. Don't see what it has to do with Vanilla. However, they will only be able to crawl what's visible to the public..... i.e. the bot can't log in.

  • Unknown
    Unknown

    More information from the customer and Botify that explains I believe more so how we might be involved. Perhaps it's a customization, perhaps it's something we can't actually do.


    Botify is a full SEO suite composed of multiples solutions:

    The first one is SiteCrawler, with which we crawl the website to discover all the pages of the website, in order to present hundreds of KPI to ours customers.

    The second one is LogAnalyzer, which allows our customers to understand how search engines crawl their site.For the first solution, we will just crawl the Vanilla forum like any other website, using our cloud crawler. For this phase, we don't need Vanilla help (unless you have aggressive blocking for bot).

    The second solution is based on the HTTP log files generated by Vanilla. We are looking at SEO visit, which means: where do visitors come from and which pages are they landing on.

    For us to get this data, we need the HTTP hit made by the visitors, stored in webserver logsfiles, with date, IP, protocol, URL, status, referer, user-agent and host if multiple host.As only Vanilla have access to theses logfiles, we need Vanilla to send these logfiles to Botify.

    The AWS bucket is here for this. But Vanilla could also send our logs to our FTP or FTPS or SFTP servers.

    cleardot.gif


  • Alex Powell
    Alex Powell
    edited February 2020

    I think we do have aggressive blocking for bots - that's part of the idea behind cloudflare too.

    As for logs, this is not something I think we share - but sounds like an Ops request.

  • Unknown
    Unknown

    Is this an Enterprise client? I can't see us sharing the http logs unless it's a dedicated cluster.

  • Unknown
    Unknown

    The deal we're talking is for Enterprise yes.

Categories