Case 07 · Cloud / Database Engineering / Backend

Cloud & Data Infrastructure

Cloud infrastructure and data storage systems

Overview

Working with cloud databases and storage: data structure, connecting backend applications, relational data, vector search and caching — each in the store that fits it.

Generalised experience across several systems

The charts are computed by this case's model on synthetic data — the same model that runs the interactive demo. They show what the approach changes, not figures from a client's system.

What I did

Designed the data structure
Backends connected to cloud databases
PostgreSQL, pgvector and Milvus
Redis as the cache
Connection configuration

Before

Mixed data: relations, search by meaning, hot reads
No single store fits all of it
Going to the database for every read is expensive

After

A store for each kind of data
Hot data served from Redis, with a limited lifetime
Connections managed through configuration, not code

Load on PostgreSQL

−70%fewer order reads reach the database

Every read from PostgreSQL100%
Cache-aside in Redis, 5-minute TTL30%

60 synthetic order reads over 15 minutes, popular orders read more often.

Average order read

2.5×faster on average with the cache

PostgreSQL by key7.0 ms
Redis first, PostgreSQL on a miss2.8 ms

The same traffic, with the demo's illustrative latencies.

What is happening

Request an order twice — the second time the cache answers. Then a list, similar documents and an expiring TTL.

The model runs in the browser on synthetic data. Times on the cards are illustrative, and motion is slowed down so the path can be followed.

Queries

Vector store

The cache in numbers

Misses0

Hit ratio—

Average order read—

Keys in Redis

The cache is empty.

Latencies are illustrative and the same on every run — the difference between the paths is what matters.

What lives where

PostgreSQL — orders, users, relations and transactions
pgvector — vectors next to relational data
Milvus — large vector collections
Redis — hot data with a limited lifetime

Engineering detailsThe full write-up: problem, architecture, implementation decisions and outcome.

Problem

Application data is mixed. Orders and users are strict relations and transactions; similar documents are found by meaning; some data is read so often that going to the database every time is expensive. No single store fits all of that, and connections to several cloud services have to be set up so they are convenient and safe to manage.

Solution

A data layer with a store for each kind of data — PostgreSQL for relational data, pgvector or Milvus for vector search, Redis for frequently read data — connected to backend applications through configuration rather than code.

My role

Designed the data structure.
Connected backend applications to cloud databases.
Worked with PostgreSQL for structured data.
Used vector stores — pgvector and Milvus.
Worked with Redis.
Set up how the components interact and managed the connection configuration.
Used cloud services.

Technical architecture

Web Client → Backend Services → data layer: PostgreSQL for relational data, pgvector or Milvus for vector search, Redis as the cache, and cloud services. The backend decides which store fits a query.

Implementation

A store for each job. PostgreSQL for relations, constraints and transactions; pgvector for vector search next to relational data; Milvus as a dedicated vector store for large collections; Redis for fast access to frequently read data.
Cache. Redis serves frequently read data without a database round trip, and a limited entry lifetime keeps the cache from drifting away from the database for long.
Configuration outside the code. Connection addresses and settings come from the environment, so one application runs in different environments without code changes.

Functional outcome

Backend applications work with cloud databases and stores, each used for its purpose: relational data, vector search and caching are kept apart, and connections are managed through configuration.

Need a system like this?

Tell me about the problem — I'll come back with questions and a proposed approach.

Discuss your project Hiring? See my engineering profile