A year or more ago on my old blog I wrote an article about how I think we could use logging information in hybrid integration scenarios to bring together this holistic view of whats happening across the stack. The old link was:
More recently Ive spent some time playing around with Azure Service Bus Event Hubs and exploring the idea of instrumenting my applications and integration components with log4net where it is configured to send logging information to Azure Service Bus Event Hubs. The aim was to implement the scenario described on the article above and now Event Hubs is GA I felt that it was the perfect place to send my logging data because of the following reasons:
- It is capable of dealing with massive volume
- It has some good ways to process data from the stream such as Stream Analytics
- Strategically I believe it will fit really well as an on ramp to the Microsoft Business Intelligence Stack
- Its easy to send data to it
If your in the .net world then log4net is one of the most common libraries you will use for logging and again about a year ago I showed how you could create a log4net appender for Azure Service Bus Messaging, so it made sense to do a new post with a log4net appender for Event Hubs.
In the project I have 2 appenders
- SBEventsAppender = A simple appender for Event Hubs
- SBEvebtsBufferedAppender = An appender where you can buffer the events and process them with multiple threads and the batch methods in the service bus SDK
The appenders are pretty easy to use and the code download below gives you a sample console application demonstrating their use and some unit/performance tests in visual studio to let you try them out too.
To give you an example of the performance I have the following test which simulates something like a real user scenario calling a component that might log 3 events and do some stuff which takes about 100ms to complete.
public void BufferedLogging()
Running this on my development machine at home, I was running the Buffered appender with 4 publishing threads using a visual studio load test with 100 concurrent users. After running this test for 10 minutes I had been averaging 960 tests per second and completed a total of 580,731 tests. This means Ill have pushed somewhere in the region of 1.7 million events to the event hub. That’s pretty good performance for most logging scenarios.
Now obviously in your application you dont want to send every single logging item to a centralized log store but for those key events (perhaps info, warn, error in the log4net world) you have a viable option for most application scenarios where you can send your log events and do multiple interesting things with them out of process.
To get the code sample Click Here
Ive been using this for a little while now and it seems to work really well but I would like to hear feedback from others to see if it suits their scenarios well too.