What should I do with my Apache access logs or Nginx access logs?
Working daily on exciting projects, our experts’ and evangelists’ team creates a deeper pool of log analytics knowledge. As we watch our partners innovate and create new use cases, we keep on developing a better view of how to extract value from log analytics, and much of it comes from access logs, Nginx or Apache access logs if it’s the case.
Even as every company has its own processes and business model, they all encounter some common challenges. And they all need to get clearer information to drive their business forward, taking the right decision at the right time, and evaluating their impact.
Access logs are standardly used and very easy to collect. Whether you’re using Apache and have Apache access logs or NGinx or any other technology to collect them, their content is the same:
A timestamp
An action
A few technical informations (status code, client browser, response time, etc.)
Apache access logs, like Nginx, contains everything you need. Logs are usually presented as raw event but you easily transform them into a structured object.
Analysing them will help you create a global and unified vision of your system that can be used both by your tech team and business departments. You’ll find below what are the best practices that you can directly get from your access logs.
I. Proactive troubleshooting for a better user experience
1) Monitor response time
Much has been written over it and it’s now common knowledge that website response time has significant impact over traffic, and sales. A 1-second delay in page or mobile app response can result in a 7% reduction in conversions. Even if you offer awesome services or otherwise incredible user experience, as your user just left, you will never be able to show it to them. You just lost a deal. Monitoring your website performance goes far beyond technical needs, it’s a complete business imperative.
Measuring your app global response time is a good start, as you can be aware of something happening. But most importantly, it’s your ability to react quickly and fix the growing latency experienced that will impede a large number of your user to have a seamless experience. You should be able to quickly spot if latency comes from the API backend, or from the number of connected users or unusual requests for example. To get these insights you should:
Prefer percentiles rather than mean aggregation. Percentile, commonly used in statistics, is a measure that indicates “the value below which a given percentage of observations in a group of observations fall”. While the mean would show you the average latency experiences, the 95th percentile would focus on the 5% worst cases. Percentiles can thus more easily show you the way to bad customer experiences and a real issue for your app.
Fine-monitor your app critical path or conversion funnel (home page, key categories, cart & payment, most viewed articles/pages, login screen, etc.)
Build response-time dashboards split by device and by country. It will allow you to quickly filter and sift through information when something is happening. Is the longer latency experienced in all countries on a specific browser or in a specific country? You should get a dashboard similar to the following one:
Once you’re done, don’t forget to create alerts in order to be aware of anything happening before social networks or support!
2) Track errors and broken links
Many companies have their app not only opened to their user but also to their suppliers and partners in order to fluidify interactions. As the number of these interactions increase, it becomes a true challenge to have a 360° vision on what is happening in your kingdom.
Errors and broken links give you a real-time feedback on the app status. A deep analysis of these errors can give you very useful hints on what’s really going on. Have a look at the following examples:
The 500 errors in your back-office are probably a telling sign of problems in your API. Check if you released recently, if one of your server is down or if you are not under DDOS attack.
All 40x errors in the back-office could be telling your partner / user is still using a recently deprecated API. With the access log, you can easily see who is using it and inform them of the correct new way to do what they’re doing, thus avoiding further 40x errors on your side.
Lastly, monitor the number of errors and their type per user or per module help the tech team to be more proactive in fixing issues as they have access to all the contextual information they need (user, server, time, module app, etc.).
And as for the monitoring of response times, getting an alert on your broken links or a deprecated API resource will make life easier for you, as you won’t have to spend time checking performance and errors.
II. Access logs provide business value…
Real-time product feedback
Most commonly used web data solutions such as Google Analytics only provide aggregated data. When you need to get more fine-grained information about a specific feature or a specific usage, the information is simply not there.
Getting information about a specific feature usage is for example very useful to validate new product developments. You can have a look at tech and business performance related to the deployment of a new feature, as a kind of post mortem, in order to adjust further developments. Or even better, you can A/B test versions and analyse performance before actually choosing which version you want to implement.
Integrating access logs in your test-cases reviews will allow you to compare 2 or 3 candidate versions and only implement the better performing one.
If you have added other user information to your access log you can collect the user session time. This last indicator may be very useful to track the gain of the new release. If you want to go deeper and know more about it, read the interesting and innovative use-case of Adikteev.
Conclusion
Like many others sources, access logs are a wealth of knowledge. Even if logs are typically sheltered within tech systems, they generate company wide value. Stay tuned for our Real User Monitoring solution (RUM) post using the ignored source of web browsers logs.
