Migrating Unstructured Customer Agreements: From Free-Text Fields to Hard Data
The foundation of a stable rollout
Every ERP or TMS migration follows a predictable pattern. The project team builds new data models, tests integrations, and schedules the cutover. And then a specific category of data emerges that refuses to be squeezed into columns: free-text fields. Memos, remark fields, and customer-specific instructions—often dozens, sometimes hundreds of thousands of text lines manually entered by employees over the years.
These fields hold operational agreements that aren’t documented anywhere else. Loading times, vehicle requirements, location-specific contacts, and seasonal exceptions. When transferred directly to a new system, these notes turn into unreadable strings that no automated workflow can interpret. The result: planners fall back on phone calls and emails, drivers end up facing closed gates, and the promised efficiency gains of the new system evaporate even before go-live.
Therefore, the foundation of a successful migration lies not in the technical infrastructure, but in the underlying data quality. To ensure the process of cleaning or migrating customer data is successful, you must first understand what is hidden in those free-text fields—and how to translate that information into hard, usable parameters.
The operational risks of a lift-and-shift strategy
A 1-to-1 migration—transferring all data completely unchanged from the old system into the new—seems like the fastest route. In practice, this approach creates blind spots that only become visible once the new system is live and automated workflows attempt to process the legacy free text.
Where operations break down:
Distribution centers and loading/unloading locations adhere to specific time windows, entry routes, and vehicle restrictions. When these instructions are buried in unstructured text fields, a TMS cannot factor them into route planning or slot booking. The result: waiting hours, rejections at the gate, and manual escalations.
Invoicing and rate calculations rely heavily on customer-specific conditions. If these conditions exist solely as free text (“discount for full loads, weekend surcharge”), the system lacks the logic to apply them automatically. Manual corrections post-invoicing consume valuable time and damage customer relationships.
Automated alerts and reporting depend entirely on structured fields. Free text bypasses every filter, KPI dashboard, and exception trigger.
In December 2021, Gartner published a press release estimating the average annual financial cost of poor data quality for organizations at $12.9 million. This figure encompasses not just direct errors, but also the time employees waste hunting down, correcting, and finding workarounds for bad data. During a system migration, these costs concentrate in a very short timeframe—exactly when an organization has the least capacity to absorb them.
A nuance is in order here: organizations that already operate seamlessly through fully standardized EDI without manual intervention experience these issues to a much lesser degree. Their customer agreements are already captured in encoded message formats. But for the majority of companies that have historically relied on free-text fields—and there are countless in transport, freight forwarding, and warehousing—the lift-and-shift route is a recipe for operational gridlock, partly due to the high risk of ERP data entry errors.
Taxonomy: Translating text into TMS parameters
The core of the solution is a taxonomy: a controlled vocabulary that translates the contents of free-text fields into defined database attributes with fixed input options. This replaces individual employee interpretations with unambiguous data points that a system can read, filter, and process.
The methodology involves three steps:
Inventory: map out the patterns present in your historical data using targeted queries. Which terms appear repeatedly? Which abbreviations are common? What categories of information can you distinguish?
Field definition: establish hard variables based on those patterns. Think of attributes like Vehicle Type, Temperature Range, Tail Lift Required (yes/no), Delivery Time Window, and Location Contact Person.
Input restriction: configure the new fields with drop-down menus and strict validation rules. This structurally eliminates free text for future data entry, preventing recurring data pollution.
Pattern recognition in raw data
Large volumes of free text cannot be scanned manually. Therefore, the first step is running targeted queries on your legacy database to identify frequent terms, acronyms, and structures.
For example, search for:
Time indicators: patterns like “before 14:00”, “not on Fridays”, “morning only”
Vehicle references: “box truck”, “reefer”, “mega trailer”, “van”
Location instructions: “rear entrance”, “dock 3”, “left via barrier”
Contact details: phone numbers, names, job titles
Conditional agreements: “provided that”, “only if”, “not in combination with”
These queries generate a frequency table that highlights which information categories appear most often. These categories form the baseline for your new field structure. According to IBM’s definition of unstructured data, this type of information accounts for the overwhelming majority of all enterprise data—and it is growing significantly faster than structured data. Without active conversion, this intelligence remains entirely unutilized in any new system.
Practical example: Dissecting free text
Take the following note from a legacy system:
“B12 via rear entrance, not Fri. 1400h, reefer mandatory”
To an experienced planner, this makes perfect sense. To a TMS, it’s a meaningless string. The translation into structured fields looks like this:
Free text fragmentTarget fieldStructured valueB12Location code / DockB12via rear entranceAccess routeRear entrancenot Fri. 1400hTime windowDelivery before 14:00 excluded on Fridayreefer mandatoryVehicle typeReefer (mandatory)
One single sentence becomes four discrete, searchable, and filterable fields. The TMS can now automatically assign a refrigerated truck, align route planning with the correct entrance, and alert the planning engine if a Friday delivery is scheduled before 14:00 for this client.
This example is relatively straightforward. In practice, notes frequently contain contradictions, outdated instructions, and internal jargon that varies by branch. This raises the question of where technology ends and human interpretation begins.
Balancing RPA and manual interpretation
Extracting structured data from free text can be partially automated—but not entirely. The most effective approach combines technology for the initial heavy lifting with human expertise for handling exceptions.
The limits of algorithms and basic NLP
Regular expressions (regex) and basic Natural Language Processing (NLP) are highly effective as an initial extraction layer. They easily recognize predictable patterns:
Timestamps in formats like “14:00”, “before 2 pm”, “morning”
Postal codes and number sequences
Common abbreviations defined in a standard dictionary
Yes/no indicators (“mandatory”, “prohibited”)
Where technology hits a wall:
Ambiguity: “Do not deliver on Friday” versus “Do not deliver before Friday”—a one-word difference completely changing the core instruction.
Sarcasm and informal language: notes like “preferably yesterday” or “the usual chaos at dock 4” contain critical operational context that no algorithm can reliably code.
Contradictory information: a field mentioning both “reefer” and “flatbed trailer”, likely due to a change that was never properly updated.
Corporate jargon: internal acronyms that differ by branch, team, or even individual employee. No language model can translate this reliably without extensive domain training.
Deloitte describes in an analysis of the convergence of AI and RPA how intelligent automation only yields true value when the underlying data is adequately structured. For unstructured free-text fields with high variability in language and context, the margin of error for purely automated extraction is far too high to base operational decisions on.
Decision tree for data extraction
The roadmap for the conversion process follows a clear path:
Automated pre-scan: regex and NLP extract all recognizable patterns (times, codes, standard terms) from the free-text fields.
Confidence assessment: each extracted data point is evaluated for ambiguity. Does the result fit perfectly within the defined taxonomy without requiring interpretation? → Move straight to automated mapping.
Exception handling (Fallout): records where the extraction yields multiple interpretations, fields conflict, or unknown terms appear, are routed into a human processing queue.
Domain expert review: human specialists cross-reference the fallout records against current sales files, customer contracts, and historical operational knowledge. They determine the correct translation and populate the structured fields manually.
Model feedback loop: patterns that domain experts encounter repeatedly are fed back into the automated pre-scan as new rules. This ensures the exception queue shrinks continuously with every iteration.
This hybrid methodology avoids the two extremes: fully manual processing (too slow and costly for large volumes) and fully automated processing (too error-prone for complex notes). The key lies in sharply defining the threshold—and moving that line as the model continually builds specific domain expertise.
Validation before final data migration
Structured data is only useful if it is accurate. Before decommissioning the legacy system, the new data architecture must prove it can flawlessly steer operational processes.
Shadow testing with operational data
The most reliable validation method is shadow testing: running the new data structure alongside the existing system using historical or current shipments. In practical terms, this requires:
Recalculating historical orders: take a representative sample of completed shipments and let the new system—fed solely with the newly structured data—recalculate the planning, routing, and invoicing. Compare the output against the actual historical execution.
Discrepancy analysis: every divergence between the original order fulfillment and the system recalculation serves as a red flag. Was the deviation caused by a source data error, an interpretation error during conversion, or a flaw in the new field setup?
Iterative correction: any discrepancies found are fed back to the conversion team. The taxonomy is refined, extraction rules are tightened, and the shadow testing is repeated until deviations fall well within acceptable margins.
Accenture highlights in a publication on intelligent data migration that organizations skipping a parallel testing phase face a significantly elevated risk of costly rollbacks post-launch. Shadow testing certainly takes time, but that initial investment pales in comparison to the operational chaos of launching a live system populated with flawed customer parameters.
Criteria for formal system freeze
The legacy system is only frozen—permanently switched to read-only status—once a strict set of conditions has been met:
Order completeness: the cleansed dataset perfectly reproduces historical order outcomes within predefined operational tolerances. Zero unexplained, open discrepancies remain.
Stakeholder sign-off: operational teams (planning, warehouse, transport) and finance departments (invoicing, contract management) have thoroughly reviewed and formally approved the shadow testing results.
Documented rollback scenario: a solid contingency plan is in place should the new environment encounter unforeseen issues in the first weeks post-go-live. This plan must involve read-only access to legacy data, not reopening the old system for active editing.
Active new input processes: the structured data entry framework, complete with strict validation rules and drop-down menus, is fully operational. Staff is thoroughly trained. Old free-text fields are permanently locked down or stripped entirely from user interfaces.
Only when all these lights turn green is completely freezing the old database justified. Rushing a cutover—usually driven by intense project pressure or looming deadlines—remains one of the most common catalysts for failed migrations.
Conclusion and next steps
Converting unstructured customer agreements into structured TMS parameters is not merely an afterthought during a migration—it fundamentally determines whether the new system can actually run your operations efficiently. Technology greatly accelerates extraction, but it only delivers operationally reliable results when base data is validated by experienced human experts. Shadow testing against historical orders provides the crucial final layer of quality control before the legacy system is safely retired.
If you want to prevent highly specific loading and unloading instructions from being lost entirely or incorrectly configured in a new system, you can outsource the process of cleaning or migrating customer data directly to DataMondial. Operating seamlessly from highly secure EU production facilities in Romania, their expert team structures complex logistics data using a compliant, highly accurate blend of automated extraction and specialized human domain expertise.
