From Raw Data to Reproducible Workflows

Prototyping Automated Data Organisation for the Stefan Zweig Digital Project 

Research data repositories are getting better and bigger, but they often enter the scene only once a dataset is “ready”: complete, curated, and packaged. In day-to-day archival digitisation, and research data management generally, the hard part often comes earlier. That earlier phase is where raw files accumulate, names and metadata change, and a lot of invisible work happens long before anything is fit for repository ingestion. 

This post reflects the expertise gained from developing a prototype at the University of Salzburg as part of the project Digital Infrastructure Austria (DHInfra.at)¹.  The goal of the prototype is to manage digitisation holdings preserved in Literaturarchiv Salzburg² and the Stefan Zweig digital project³. To this end, we transferred experiences from the development of the Austrian NeuroCloud (ANC)⁴.  

The ANC is a FAIR-enabling repository hosted at the University of Salzburg designed to support sustainable research data management in cognitive neuroscience. The ANC was originally developed as a digitization project funded by the Austrian Federal Ministry of Education, Science and Research⁵. In this project, we learned how much robustness you gain when data operations become explicit, versioned, and automated.  

This blog post is an excellent illustration of how projects from disciplines as far apart in theory, methodology and practice as neuroscience and the arts and humanities meet when faced with challenges that turn up when working with digital data. 

The Stefan Zweig prototype treats the early phase of digital archival work as a first-class citizen: It supports raw data organisation plus automation during acquisition, not only preservation at the end. The prototype itself and the insights gained from its implementation contribute to an important objective of DhInfra.at, that is, to build a shared and sustainable research infrastructure for the Digital Humanities in Austria. 

There are two companion posts on this blog that explain the infrastructure framing, and we strongly recommend reading alongside this one: 

Karoline Döring introduces in her article “Die Integration komplexer wissenschaftlicher Softwaresysteme in bestehende Forschungsinfrastrukturen”⁶ the overall picture and allows to relate the content of this blog post among the other DHInfra.at test cases undertaken at the University of Salzburg. 

Martin Schäler explains in his article “Die DHInfra.at-Forschungsinfrastruktur der Universität Salzburg”⁷ the kind of infrastructure at our university intended to host such services sustainably. 

Real-world bottlenecks in raw data organisation 

The Stefan Zweig digital project collects and digitises originals connected to Stefan Zweig’s life and work aiming at making their digital facsimiles available to the public. Over time, the volume of raw image material grows fast; storage fills up, but the more subtle pressure comes from the slow, but continuous evolution of the digitization process: where do files live while they are still being produced, corrected, transformed, and prepared for downstream systems? How do we ensure explainability and repeatability of all digitized artifacts without the need for excessive (manual) documentation? 

Looking at the workflow end-to-end, we quickly realised that digitisation is not just photographing objects. A significant portion of time is spent on repeated, manual coordination work: renaming files according to conventions; editing and correcting descriptive metadata; converting metadata into other formats; and assembling repository-ready submission packages (for example for the GAMS repository⁸). That manual work is not only slow—it is fragile. Errors are easy to introduce, and hard to detect once files have moved on to later stages. 

This is exactly the kind of problem software engineering learned to handle decades ago. Software teams assume that digital objects evolve; they design for constant change by keeping history, making changes traceable, and automating repeatable steps. With the ANC we learned that these practices translate surprisingly well to data acquisition processes—especially when the goal is not merely storage, but operations that can be continuously validated along a lifecycle.  

So, the guiding question for the Stefan Zweig prototype became: what if we treat each digitised object as a working unit with its own traceable history and its own automated pipeline—long before anything reaches a final repository? 

Turning a facsimile into a project 

The Stefan Zweig prototype explicitly explores partially automated curation workflows and asks how data acquisition and target publications infrastructures can be connected. Our prototype focused on the earliest segment of that chain: raw facsimile organisation and automated preparation of a submission-ready package. 

The core design decision is simple: each facsimile (more precisely, a digitised original document) becomes one GitLab project⁹. Once raw files and a metadata file are added to that project, the rest should be repeatable and machine-executable. 

Here is the practical effect of that decision, from a user’s point of view: 

When a new facsimile is added, the repository is no longer just a folder on some storage. It is a workspace with rules. GitLab gives a structured place to keep the raw material (camera files plus the metadata that describes them), and it gives you a CI/CD environment (continuous integration and continuous delivery) that can run the same data operations every time the content changes. Nevertheless, using GitLab is simple: It can be accessed via a browser, or one clones a copy of the GitLab project to their own computer, modifies the files, and pushes the result back to the GitLab server via a simple interface. 

Concretely, the pipeline was designed so that after adding the raw material, GitLab CI/CD can: 

Convert formats automatically. Raw camera files (plus their development instructions) are stored in camera-dependent proprietary format, which most image-processing software nor browsers can display. To this end, the GitLab project contains a pipeline running a command-line conversion step to produce standardised derivatives (for example JPEGs) with consistent naming. This is where “borrowing from software engineering” becomes very concrete: the conversion is not a one-off export done on someone’s machine; it becomes an automatic and reproducible build step. 

Transform metadata for downstream systems. A second automated step converts the descriptive metadata into a format expected by the target repository workflow (in this case, a METS file). The important part is not the specific schema, i.e., that we use METS. It is the fact that content and creation is well-defined, such that the transformation can be scripted, and therefore it can be versioned and re-runnable. It moreover simplifies the metadata generation for the user. 

Assemble a submission package and publish it as an artifact. Instead of manually collecting files into a deliverable directory structure, the pipeline prepares a clean package and publishes it as a CI/CD artifact. That means the ingestion package is produced deterministically from the raw sources, and anyone can download exactly what the pipeline produced. 

Generate a lightweight object website. Finally, GitLab Pages publish a small representative website for each object: an overview of the facsimile metadata, previews of the derived images, and links back to the project and to the generated package artifact. The result is a human-facing entry point that is automatically kept in sync with the underlying raw data and metadata, without building a separate bespoke web application first. 

Two prototype choices helped reduce the human error in practice. First, files inside the project can remain identifier-agnostic (simple structured naming), while the project identity itself provides the identifier—so the pipeline can insert that identifier consistently wherever needed. Second, because everything runs in CI/CD, the pipeline output becomes an early warning system: if a metadata change breaks a transformation, you notice it immediately, at the point of work, not weeks later at ingest time. 

The automation prerequisite. Automation in practice depends less on abstract workflow design and more on a very concrete constraint: the software involved must be able to run unattended. In digitisation workflows, it is common to rely on commercial tools with excellent graphical user interfaces, for example, in image development. These tools work well as long as a human is present to “click the right buttons”. In a server-based environment, however, this model breaks down. Automation CI/CD pipelines run without a graphical interface and without human interaction. Many widely used tools do not meet this requirement: they lack a stable command-line interface or depend on interactive steps that cannot be scripted. This constraint turns software choice into an infrastructural decision. Selecting tools that support headless execution is what makes the difference between “this could be automated in theory” and “this runs automatically every time an object changes”. 

What this approach solves—and what we should do next 

Using GitLab as a raw data operations backend is not about forcing data from the arts and humanities into a developer mould. It is about using a mature platform that already solves problems we repeatedly face in digitisation work. It gives you history and accountability: What changed, when, why, and by whom? It encourages structured planning and documentation alongside the data, not in separate places. It focuses on automation through CI/CD, which is where repeatable steps belong. And when GitLab is self-hosted, it addresses the very practical concern of data sovereignty: This means the repository owner has the control where the raw material lives, and under whose operational control.  

At the same time, a prototype is only a prototype. The most valuable next steps are the ones that turn it into an everyday workflow: 

• A project template so new facsimiles start correctly every time (structure, expected files, minimal documentation). 

• Validation steps that check metadata early and fail loudly (so errors are discovered where they are created). 

• A defined handover point to repository ingest (e.g., how the artifact becomes a submission package in an institutional workflow) and possibly automation of that process. 

Finally, there is a bigger, strategic idea behind all of this: a repository does not have to begin at publication time. 

Once each object has a structured project, automated derivatives, and a machine-produced package, you can imagine building a thin repository layer on top: indexing object websites, enabling search across raw and derived materials, and connecting identifiers to downstream catalogue and repository systems. That would not replace a domain-specific repository. Rather, it would make the earlier phase—raw acquisition and preparation—more visible, queryable, and robust. 

This is where the infrastructure discussion matters: Martin Schäler outlines in his blopgpost “Die DHInfra.at-Forschungsinfrastruktur der Universität Salzburg”¹⁰ the kind of virtualised cluster environment intended to host services sustainably—exactly what you need when prototypes become shared services used by many people, not just experiments on someone’s workstation.  

The main recommendation, based on this experience, is to treat automation as a design discipline: start from the repeatable steps, choose tooling that can run headlessly, and give raw data a structured home early. Efficiency is a nice side effect, but the real gain is reliability: fewer silent errors, clearer provenance, and workflows that can scale.  

---

See on this topic also on the blog:

• • Karoline Döring, Die Integration komplexer wissenschaftlicher Softwaresysteme in bestehende Forschungsinfrastrukturen. Digital Humanities an der Universität Salzburg (2025), https://doi.org/10.58079/15cfw
  • Martin Schäler, Die DHInfra.at-Forschungsinfrastruktur der Universität Salzburg. Digital Humanities an der Universität Salzburg (2025), https://doi.org/10.58079/15dki
  • Katharina Zeppezauer-Wachauer und Clemens Hafner, Die Mittelhochdeutsche Begriffsdatenbank (MHDBDB) aus infrastruktureller Perspektive. Digital Humanities an der Universität Salzburg (2025), https://doi.org/10.58079/15emt

---

Beitragsbild: TheDigitalArtist, Datenzentrum – Rechner, 2023, Pixabay Inhaltslizenz

---

 

---

Nur der Text ist unter der Lizenz Creative Commons Namensnennung-Share Alike 4.0 International nutzbar. Alle anderen Elemente (Abbildungen, importierte Anhänge) sind „Alle Rechte vorbehalten“, sofern nicht anders angegeben.

---

---