I worked on a financial application recently where there were importing data from 8 different databases nightly into a single database. The data became available at different points during the night and had lots of detailed processing around each element. And they never knew how much data was coming in. Plus there was processing for different business segments that had to be done too. There current solution took over 12 hours each night.
After lots of analysis we came up with a solution. The basic concept was to create a windows service that could be installed on any number of servers. The service had the intelligence to read an assignment table and figure out which jobs and which business segments it should operate on. Then it would use .Nets Reflection feature (Assembly.LoadFrom) to load the job. And the service was multi threaded, allowing for each job to define how many threads could run to handle it.
The prototype was up and running in just a few days.
I have used Windows Services like this for many years. The ability to use Assembly.LoadFrom is awesome and allows you to quickly build a service that can load n-number of assemblies on the fly.
