Few days ago I had to solve a problem: we had a service that published some data, that was generated in our system to another third party website. For a while, one service could cope with the publishing process, which involved few HTTP requests to other sites, database procedures and error handling. But our capacity grew and one service was not enough.
The first solution was to create a new service and run the few processes in multiple threads. It was quite efficient, but when it crashed (and it did..), we were stuck. We tried to solve the new problems , but it was not satisfactory.
So we tried a new approach, one that will help us scale better: we designed a service that will act like a dispatcher: It will collect all the new data items. The second part of the solution was a set of simple one threaded publishers, that will ask the dispatcher for a limited amount of items, publish them and ask for new ones, and so on.
The new approach, simple and stable, was perfect – we already knew that a single threaded process is stable and proven in production. This way we can increase the number of publishing processes in rush time, and decrease when not needed.
We had one issue to solve – we didn’t want to publish the same data item twice, it was crucial that it will be published once. Because of some limits, the only way to determine if an item was published was the dispatcher. So to accomplish that, I had to design it in a way that it send an item to the publishers only once.
This task was fairly easy, and this the way I achieved it:
- The dispatcher is a singleton.
- The dispatcher has a queue of data items.
- The dispatcher has a repeating process, running in a different thread that query the data items source for new items, and enqueue them.
- Each enqueue and dequeue of item involves locking the queue, to prevent doubling.
- Publishers the request data items, receive a small number of new items, thus able to publish them quickly and get new ones.
I implemented the dispatcher as a WCF class library, hosted in a website on IIS. The publishers are small WinForm apps (just for the GUI) but it could be a console app or a windows service. Scaling is easy (to some extent): We can multiply the number of publishers, as well as the number of dispatchers.
This is it, simple, scalable and stable solution.
Happy coding :)