Case 02 · Data Engineering / Scientific Computing / Python

Scientific Data Processing

Numerical processing, interpolation and visualisation of data

Overview

Software tools for numerical data: preparing datasets, numerical methods and interpolation, and plots that compare the original and the processed values.

Generalised experience across several systems

The charts are computed by this case's model on synthetic data — the same model that runs the interactive demo. They show what the approach changes, not figures from a client's system.

What I did

Import and preparation tools in Python
Cleaning and transformation with Pandas
Interpolation with NumPy and SciPy
Comparison plots with Matplotlib

Before

Gaps and outliers in the measurements
Readings taken at a step other than the one needed
Hard to check how processed values relate to the original

After

Data is cleaned before any computation
Values in between are restored by a method chosen for the data
Plots compare the processed values with the original ones

Error after cleaning

12×smaller error once the gaps and the spike are handled

Raw readings, spike kept0.31
Cleaned readings0.025

A sensor recording with gaps and a spike, PCHIP interpolation, RMSE against the true signal.

Choosing the method

108×smaller worst-case error than a high-degree polynomial

Polynomial through all points1.9
Cubic spline0.022
PCHIP0.018

The Runge function from 11 even points; the largest deviation from the true curve.

What is happening

Pick the data and a method, move the points, then run the pipeline — the data goes through Pandas, NumPy, SciPy and Matplotlib.

The model runs in the browser on synthetic data. Times on the cards are illustrative, and motion is slowed down so the path can be followed.

Data and method

Dataset

A high-degree polynomial through evenly spaced points swings near the ends — Runge's phenomenon. The spline stays stable.

Points: 11

Interpolation method

Plot

Original function
Lagrange
Points

Drag the points with a mouse or a finger, or move them with the ↑ ↓ keys.

Methods compared

Interpolation method	Max error	LOO RMSE
	0.067	0.18
	0.022	0.16
	0.018	0.18
	1.9	4

The data is synthetic: the original function is known, so the error can be measured honestly. LOO predicts each point in turn from the others.

Engineering detailsThe full write-up: problem, architecture, implementation decisions and outcome.

Problem

Numerical data rarely arrives ready for analysis: values are missing, outliers occur, and measurements are taken at a step that differs from the one needed. To compute, compare and draw conclusions, the data has to be prepared, the values in between restored, and the result presented so that it can be checked.

Solution

Python processing tools built as a pipeline of separate steps — loading, cleaning, computation and interpolation, plots — so each step can be checked on its own and the interpolation method can be chosen to fit the data.

My role

Built Python tools for importing and preparing datasets.
Implemented data cleaning and transformation with Pandas.
Worked with NumPy arrays and numerical methods.
Implemented numerical processing with SciPy's interpolation methods.
Built Matplotlib plots that compare the original and the processed values.
Prepared the results of the analysis.

Technical architecture

Processing is a pipeline of separate steps: loading → cleaning → arrays → computation and interpolation → visualisation → report. Each step takes and returns data in a clear format, so it can be checked and replaced independently of the others.

Implementation

Preparing the data. Gaps and outliers are handled before any computation: interpolating dirty data reproduces the measurement errors.
Choosing the method. Linear interpolation is simple and predictable, a cubic spline gives a smooth curve, monotone PCHIP adds no false oscillations at sharp transitions, and a high-degree polynomial on an even grid swings near the ends — Runge's phenomenon.
Checking the result. Original and restored values are compared on plots and by error metrics.
Working with arrays. Computations run over whole NumPy arrays.

Functional outcome

The tools load a dataset, clean it, restore the values in between with the chosen method and produce plots that show how the processed values relate to the original ones.

Need a system like this?

Tell me about the problem — I'll come back with questions and a proposed approach.

Discuss your project Hiring? See my engineering profile