Bridging the Gap: Utilizing Multi-Modal Learning to Decode Medieval Herbals

Introduction

For centuries, medieval herbals—manuscripts detailing the medical, culinary, and magical properties of plants—have remained enigmatic treasures. These codices are characterized by a distinct “multi-modal” tension: the interplay between stylized, often abstract botanical illustrations and dense, archaic textual descriptions. For historians, linguists, and digital humanists, the challenge lies in the fact that these two streams of information rarely align perfectly. A plant may be depicted with a vibrant, non-naturalistic color palette, while the Latin or vernacular text describes it through the lens of humoral theory.

By applying modern multi-modal machine learning (ML) architectures, we can bridge this historical divide. We are no longer limited to analyzing text and images in silos. Instead, we can synthesize these signals to map medieval symbolic logic, trace the migration of botanical knowledge across cultures, and even identify anonymous illustrations. This article explores how you can leverage multi-modal learning to unlock the secrets hidden within the pages of medieval herbals.

Key Concepts

To understand the application of multi-modal learning in this context, we must first define the core pillars of the technology:

• Feature Extraction: In medieval herbals, this involves converting visual symbols—such as the serration of a leaf or the orientation of a root—into mathematical vectors using Convolutional Neural Networks (CNNs). Simultaneously, text is processed through Natural Language Processing (NLP) models to extract semantic embeddings from historical descriptions.
• Contrastive Learning: This is the backbone of modern multi-modal models like CLIP (Contrastive Language-Image Pre-training). It trains a model to understand that a specific sketch of “Mandragora” belongs to the descriptive text discussing its soporific properties. By maximizing the similarity between the correct image-text pair, the model “learns” the symbology of the period.
• Cross-Modal Alignment: This refers to the ability of the system to map features from the visual domain to the linguistic domain. If a medieval illustrator used a specific geometric pattern to denote “toxicity,” a cross-modal model can learn to associate that visual motif with textual warnings like “cavete” (beware) or “mortiferum” (deadly).

Step-by-Step Guide

1. Digitization and Annotation: You cannot train a model on “fuzzy” historical data. Begin by sourcing high-resolution scans from repositories like the British Library or the Wellcome Collection. Annotate your data by pairing specific image regions (bounding boxes around the plant) with their corresponding textual snippets. Ensure you have a ground-truth dataset that defines which symbols correspond to which descriptors.
2. Preprocessing Historical Corpora: Medieval texts are notoriously difficult due to non-standardized orthography and Latin abbreviations. Use an Optical Character Recognition (OCR) engine specialized in historical scripts, such as Transkribus, before feeding the text into your model. Normalize the language to a standard medieval Latin or the relevant vernacular to reduce noise.
3. Architectural Setup: Implement a dual-encoder architecture. Use a Vision Transformer (ViT) to process the images and a transformer-based encoder (like RoBERTa or a multilingual BERT) for the text. By connecting these via a projection head, you can create a shared embedding space where images and text can be compared directly.
4. Training for Symbolic Correlation: Train your model using a contrastive loss function. The goal is for the model to successfully retrieve the correct textual description when presented with a specific illustration, even if that illustration is highly stylized or “mythical” in appearance.
5. Verification and Iteration: Evaluate the model using “zero-shot” retrieval tasks. Present the model with a plant illustration it hasn’t seen before and ask it to predict the textual description. Refine the model by introducing human-in-the-loop validation, where subject matter experts weigh in on the model’s association accuracy.

Examples and Case Studies

The “Herbarium Apuleii” Mapping: Researchers have used multi-modal approaches to compare the Herbarium Apuleii Platonici across different manuscripts. By synthesizing the visual data (how the plant is drawn) with the text (the properties assigned to the plant), the model revealed that illustrations often evolved faster than the textual descriptions. This provided concrete evidence of “iconographic drift,” where illustrators copied images from memory rather than observing the plant directly.

Identifying Mythical Flora: Many medieval herbals contain plants that do not exist in the biological record. By using multi-modal learning, analysts were able to cross-reference these “impossible” plants with text describing their symbolic virtues. The model successfully identified that certain visual symbols (e.g., specific vine shapes) were linked to religious iconography, confirming that the plants were symbolic constructs rather than failed botanical attempts.

“The integration of visual symbols and textual descriptions does not merely translate the manuscript; it reconstructs the cognitive framework of the medieval scribe, revealing a world where botany was as much a branch of theology as it was of medicine.”

Common Mistakes

• Overfitting to Modern Taxonomy: A common error is forcing medieval plants into modern Linnaean categories. Medieval authors grouped plants by their “virtues” (heating, cooling, drying) rather than their genetic relationships. If your model forces a modern taxonomy, you will lose the historical nuance.
• Ignoring Stylistic Variation: Medieval art varies wildly by region and monastic tradition. A style used in a 12th-century English scriptorium will differ from a 14th-century Italian workshop. Failing to account for this leads to poor cross-manuscript generalization.
• Neglecting Scribal Abbreviations: Medieval Latin is full of tildes, ligatures, and contractions. If your text encoder treats these as noise, you lose critical keywords. Always use an OCR/NLP pipeline designed specifically for paleography.

Advanced Tips

To take your multi-modal synthesis to the next level, consider Attention Map Visualization. When your model identifies a plant based on a text description, use “Grad-CAM” or similar techniques to generate heatmaps. These will show exactly which parts of the illustration the model focused on to make its decision—for example, the shape of the leaf or the type of root. This effectively turns the black box of machine learning into a tool for art historical discovery.

Furthermore, integrate Contextual Metadata. Incorporate the geographic origin and the date of the manuscript as auxiliary inputs into your model. This allows the system to learn how botanical knowledge evolved over space and time, essentially building a “spatiotemporal map” of medieval knowledge. You can then query the model: “Show me how the visual representation of Artemisia changed from 1000 AD to 1400 AD as the textual description shifted from Greek-influenced to Arabic-influenced medicine.”

Conclusion

Utilizing multi-modal learning to synthesize visual and textual data in medieval herbals represents a major leap forward for digital humanities. We are no longer limited to reading the text or merely looking at the pictures; we can now understand the deep, logical connection between the two. By building models that respect the unique, non-naturalistic, and highly symbolic nature of these manuscripts, we can decode how medieval thinkers conceptualized the natural world.

To succeed in this endeavor, focus on high-quality annotation, account for the stylistic quirks of medieval paleography, and ensure your model architecture respects the symbolic—rather than the strictly biological—intent of the original author. As these models become more sophisticated, they will not only provide insights into history but also demonstrate how multi-modal AI can preserve and interpret our most complex cultural heritage.