TRUNCATE making replicated MySQL table inconsistent

Here at TIM Group we make use of MySQL’s statement-based replication. This means that some functions, like UUID and LOAD_FILE, cannot be used when we write code or do manual maintenance because they break the consistency of the slaves. We now know we have to add the TRUNCATE statement to the list of things not to use. This is how it went…

One of the applications we have is used for background jobs: you send it an HTTP request to have a job started and then you look at its database for the results.

We needed the result of running one of these jobs for some analysis. The instructions looked like:

1) TRUNCATE table job_results
2) send request to /jobs/new
3) extract content of job_results table

We started following the instructions but after a few minutes of staring at the terminal waiting for the request to complete we thought that the request had a problem. We hit Ctrl-C, killing the request, ran TRUNCATE job_results again and restarted the request.

Our second attempt was faster to respond: after a few minutes it returned an error.

And after a few more minutes the two slave databases started reporting an error with statement-based replication. They could not insert a record in the table job_results because the primary key was duplicate.

Continue reading

From structure to behaviour – migrating to an Event Sourced system

For a few years now we have identified Event Sourcing as being a good fit for the sort of applications we build at TIM Group. This is becoming increasingly important as the business has been transitioning from an alpha capture and distribution tool to a platform that provides analytics as well.

IdeasFX, one of our newer initiatives, was exhibiting various issues even in its infancy. Although we had some stability and scalability issues, the largest negative impact on the team was being cited as the very confused “domain model”. The impact came in the form of slower development cycles caused by a large amount of accidental complexity and low team morale for the same reason. Although the product was not making much money and almost certainly could have continued as it was, we needed a product to demonstrate the value of a domain centric Event Sourced approach so we could later apply it to more of our core products.

So the team went to work. We started by researching CQRS/Event Sourcing topics: we bought Greg Young’s and some of Udi Dahan’s online courses, we read books on Domain Driven Design and Event Sourcing. We were lucky enough to have Greg in for a couple of days during our technology summit to help us clarify our thoughts.

Finally, we started a series of modeling sessions to try and formulate the concepts we were working with. These sessions turned out to be invaluable, we were lucky enough to have product people with us who truly understood the domain and were able to help the team reach a shared-understanding of the logical system and ubiquitous language.

Shortly after these initial sessions, we begun work to improve the system!

We intend to publish a series of posts explaining techniques we are using and our reasoning for taking the approaches we did, starting with the “Event Store Adapter”.