A short history about SQL Server scalability

As more and more of our customers communicate with us in English, I decided to start blogging in English.

So for my "new" readers here is a small introduction to what I do with SQL Server and other similar technologies. Originally I'm a developer, who is gradually immersed in various database technologies. I have been working with databases for more than 30 years now. I started my own business with my partners in 1994. Then, in the late 1990s, I discovered the vast possibilities of Microsoft SQL Server for our customers. Our office is in Hamburg, and since 2017 we also have an office in Karlsruhe in the south of Germany.

The company name "Projekt Service" expresses what we do most of the time. We support projects of any kind, as long as they are related to SQL Server or databases in general, with our products. A plan can actually mean the classic long-running project as well as a short-term action to eliminate unwanted impact in development or operation.

What value do we bring to projects? What are our products? As we have been here for several decades, our services have already undergone some changes. For a long time, we have only been working as consultants in the field of project work. About 10 years ago, we started to train and coach our customers to gain a deeper insight into the technical aspects of SQL Server. Our customers were enthusiastic about the technical depth and the many details that we passed on to their employees during the training sessions. Over time, we developed the training courses further together with our customers. The wish of the companies was that we not only provide technical expertise but also integrate methods and processes to apply this knowledge. This is how we developed our two methods PSG PDD & PSG RDD.

With PSG PDD (Performance Driven Development) we address the development process of software solutions about the software properties performance and scalability. By this, we do not mean the usual subsequent performance tuning, but an acceptable method for mastering these two topics at every stage of development. The PSG PDD method addresses not only the real developer but also the team leader as well as the project manager, product owner and stakeholder.

We give the necessary knowledge to apply the method to organizations, in several training modules and maturity levels. An essential aspect of this is that we want to avoid so-called rock stars in teams. Therefore, we do not provide much more in-depth technical knowledge at the higher levels, but rather the capabilities to measure and maintain performance and scalability and return these measurements as feedback to the team. They are more of training to be able to take responsibility for these areas and to better support colleagues in this aspect.

With the PSG RDD (Repository Driven Development) method, we contribute our experience in managing very complex and large environments.  Examples can be environments with several hundred servers or ETL processes with many dynamic sources distributed all over the world.

A noteworthy example for one of our first RDD implementations was a SQL Server Replications environment with over 1000 servers, which represented a global production control. Microsoft asked us to make this (maybe a bit questionable) architecture more manageable for the customer. The client's team had tried to tame this beast using a series of scripts. But there were too many heads, no matter how many scripts they created. So we decided to create a multi-level repository around the replication topology of the SQL Server to manage these enormous structures much more efficiently and automatically. In the following years, we have built various repositories around technologies such as Service Broker and CDC to make them even more efficient and manageable in complex environments.

We are also often consulted to evaluate the status of an ISV's development or product concerning the use of SQL Server's capabilities and best practices. A frequent cause for such engagements are complications with our core subjects performance and scalability. By our evaluation, concrete measures are then arranged with our customer to address the problems. It is usually agreed that we should train the whole development team or at least some members.

Frequently, we were also called upon to create monitoring concepts for SQL Server environments. Our primary concern is not so much the selection of a product but rather the design of monitoring processes which have maximum effectiveness about the possible actions. What is the advantage of an administrator if he or she continually receives information in monitoring about what to change in the software if he or she is not allowed to make these changes? On the other hand, an administrator who works with a team of developers should be able to reconcile the developer's expectations regarding the behavior of the solution with the state of the production environment at any time. In some cases, customers have also decided to delegate parts of the operational responsibility to us.

While this has been my professional background over the last years. I have presented my experience on these topics at various community events in the UK and Germany. Since 2005 I am responsible for a local SQL Server community in Hamburg, which is a chapter of the PASS. I received the MVP Award for my community work in 2014, 2015 and 2016.

When I look at the topics of my talks in recent years, my progress is quite clear. I've always been a developer and ever tried to automate processes. Over time, in addition to my enthusiasm for technology, I have also attached increasing importance to the efficiency and cost-effectiveness of my recommendations and activities.

My blog thus reflects my current interests:

Helping customers heal slow systems.
The transfer of know-how to design systems fast and scalable. These critical properties of a solution are challenging to achieve afterward.
Bringing new, currently available technologies such as my favorite in-memory OLTP to companies.
And, last but not least, to prepare customers for new and fundamental other ways to develop their solutions so that they are ready for the effective use of cloud infrastructures, Docker, etc.

I would be pleased if you would follow me on my journey through the Data world.