What was the client's problem?

The original problem was the high number of different data sources as well as the different formats that made up the basis of the daily work. In addition, over the course of the last few years, not only the data and its structures, but also the internal, manual workflows have grown historically.

The challenges were therefore the following:

• The data had to be analysed and restructured.
• The manual workflows had to be analysed and understood by our staff.
• The manual workflows had to be redesigned and reprogrammed.
• Quality assurance had to be ensured at all times because operations had to continue.

Another challenge was the management of the project due to the local distance between the customer in Arnstadt [Thuringia] and parsQube in Karlsruhe [Baden-Wuerttemberg]. This meant that both the exchange of requirements, the development as well as the acceptance of milestones and the acceptance of the finished releases had to be redesigned and implemented.

Why was parsQube asked?

One of the main data sources was generally in XML format. Specifically, it was the database system eXist-db, which was developed by Wolfgang Meier in 2000. In addition to eXist Solutions GmbH in Lenzkirch, which offers support and services for exist-db, parsQube, with its many years of experience in working with eXist-db, is one of the few eXist-db experts in Germany.

Our contact person thus very quickly became aware of us during the research, as we have been dealing intensively with data analysis since 2013. In addition, we can provide references on numerous large-scale XML projects (see our references).

How did we proceed methodically?

Communication

Our communication with the client was based on three pillars:

• Personal meeting at the customer's site for a joint kick-off.
• Regular telephone consultations between the customer and the development team.
• A common ticket system to manage the requirements and to record the telephone consultations and the commitments made.

Planning and procedure

Together with the client, we agreed on an iterative procedure. The individual steps of this procedure are explained in more detail below:

The starting point was a written specification submitted by the client in the form of a pdf, which described all functional and non-functional requirements of the desired product as well as a prioritisation of these requirements. Prioritisation and specification formed the basis for our internal planning rounds.

In the planning sessions of our developers, the requirements desired by the customer were reviewed and discussed in detail. Initial ideas for implementation were exchanged, potential challenges identified and possible ambiguities in the specification made transparent. Subsequently, the requirements were entered into the ticket system as tickets and assigned to a so-called target version (a digital backlog that contained all of the customer's desired requirements). Last but not least, an estimate of the effort was made.

In the next step, we arranged a telephone appointment with our contact person on the client side. The tickets were reviewed via the common ticket system and any ambiguities identified in the planning round were clarified. In addition, the tickets were occasionally further detailed at this point in the form of (additional) acceptance criteria, screenshots or more precise wording. The joint coordination with the customer thus created the perfect basis for the subsequent implementation of the ordered tickets. To ensure absolute transparency, discussed points, results, commitments and agreed deadlines were also logged in a wiki in our ticket system.

Once the target version had been delivered, a telephone appointment was made for acceptance. In this context, the client was able to give us live feedback. Then the process started again: with the specification and a prioritisation. This procedure allowed for a flexible and accurate implementation of the customer's requirements.

What was the idea/concept?

The first challenge was to cope with

• the high number of data sources and
• the heterogeneous "data landscape" (data formats)

The idea was to create a kind of "data hub" that transfers all data into a uniform data format. Since the client was already working with an XML database, the idea of choosing an XML data format as a basis was very obvious.

During the conversion of various data formats into the uniform data format (data transformations with XQuery and XSLT), we introduced new process steps that performed quality assurance of the overall data during the data transformation with the help of XML Schema & Schematron. Here, attention was paid to possible redundancy of the data, but also to the plausibility of the overall data.

How was the idea/concept implemented?

The data importer was written in Python. In addition, an XML database (eXist-db) and an SQL database (Maria-db) were used in the project. The backend of the analytics application was programmed in XQuery and the frontend was built with HTML and TypeScript.

If you have any further questions about the technical implementation, please do not hesitate to contact us. We will not reveal any secrets of the client, but will answer your questions regarding the use of the technologies. Feel free to call us.

See our team page.

back