DB Logger: pruning & enabled over long periods of time
We've had multiple discussions over time about this plugin. To make a long story short, once this plugin is enabled, it logs an enormous amount of data in the GDN_EventLog table until it is disabled again. We do have some pruning in place but only on data older than 90 days.
We currently have 578 instances (about 133 if we exclude Adobe) running this plugin even though in theory this plugin is meant for debugging purposes, I do think it is forgotten enabled in a lot of cases.
We have seen over time the GDN_EventLog table reach multiple gigs in size.
We have discussed mainly 2 options on how to handle this, perhaps there is more options but these are the 2 that came up before.
- Shortening the time window before the data is pruned. 90 days (our current default) becomes very long when you take into consideration the amount of data being stored.
- Automatically disabling the plugin after a certain time window.
Both options would be configurable to something else when needed.
Fact to keep in mind, Adobe runs this plugins on all of their instances as a requirement.
The goal here is basically to gather opinions on this "issue". I know @Todd had some concerns.
Comments
-
Is there a good reason it's enabled even on those 133 instances (or even on Adobe for that matter). I imagine for Adobes it's in their node config. Maybe I should i just set it to false in the node config?
Basically, if anyone comes across this addon enabled and we're not in the process of debugging anything - should it be disabled?
0 -
I would first confirm with them before turning in to false everywhere. It was mention to me that it was perhaps something they required.
And yes this was always my assumption, it's kind of one of the reason I made the post.
0 -
I just had to spend hours trying to prune Acer's logger which had over 10 million rows just so that I could export it and do some research.
I know that one solution would be to turn logging off on some of these hi traffic sights but you can't always catch the data you need after something has gone wrong.
Maybe to keep the logs down we could only log certain levels of events ( like < 4) and then maybe clean up our code. Also the analytics tool writes a ton of rows to the log. Maybe that could be less.
0 -
One thing I think that needs to be clarified is - when Joe sells Audit Logging to our Enterprise plan, is that the same thing as the event log? I think so. This is how some of these new Enterprise deals get into the 6-8k monthly range. We can't just be turning it off if it's being sold.
2 -
One idea would be to create a tiered system of what events are logged. Maybe a "normal" mode for folks who pay for it and a "debug" mode that does more when we need it. We could go further and upsell more granular logging if we can figure out how to draw that line.
0 -
That issue brought this topic back up https://github.com/vanillaforums/estimates/issues/563
I think both suggestions made above weren't working for Todd.
Shortening the time window before the data is pruned. 90 days (our current default) becomes very long when you take into consideration the amount of data being stored.
Automatically disabling the plugin after a certain time window.
Suggestion 2 wasn't good because we sell that plugin to customers as audit login.
Suggestion 1, I can't remember the specifics exactly but I don't think we wanted to shorten that timeframe.
What we currently log in there, and what should we really log in there? Should what we log in there be configurable/customizable?
I don't think I'm the right person to answer that question but for example Acer (I used Acer because it's the customer mentioned in his post above but it could be any other of them) has that plugin enabled on a permanent basis. From that they have about 10M records in that table at any given time which means about 5G of data. Right now they have 9,650,761 records, but if we remove all analytics_menu event it goes down to 2,681,894. Cutting down ours logs from the DB logger by over 70% in that case. Reviewing what we need/want to log in there to answer the requirements of Audit Logging and the internal need we make of that plugin could be a first step in the right direction. previous suggestion kind of work in the same direction and could potentially be the answer.
One idea would be to create a tiered system of what events are logged. Maybe a "normal" mode for folks who pay for it and a "debug" mode that does more when we need it. We could go further and upsell more granular logging if we can figure out how to draw that line.
We have a dashboard search ...
Technically we have a dashboard search coming with that plugin, and could have came in handy for golfwrx, enabling them to search exactly what they need from those logs (after we added what they needed to log to it), but that plugin logs so much data I doubt this search could ever be used anywhere in any performant way. Which kind of kills any attempt to answer use cases where we would want to answer a customer need for information with that plugin, not even sure if we want/should go down that road, especially with the current state of that plugin.
0 -
0
