Building a chat app with S3 object storage: what S3 does, what it doesn't, and how to wire it

The short answer: yes, but S3 is one of three layers

A chat app does three different jobs. It delivers each new message to everyone in the room the moment it is sent. It remembers messages, so a user who opens the room sees the last few dozen, and the app can answer questions like "what is unread". And it stores files: photos, short videos, voice notes and documents that people drop into the conversation.

S3-type object storage is very good at the third job and not built for the first two. So the honest answer to "can I build a chat app on S3?" is yes, with S3 as the file and archive layer, next to two other pieces:

An app server that keeps a WebSocket connection open to every client and fans out messages. It is small: a few hundred lines of Node.js with Socket.io is enough for a working chat.

A database that stores messages as rows, so "last 50 in this room" is one indexed query.

Object storage for everything that is a file. It keeps the database and the app server light, because neither of them has to hold or move megabytes of media.

The rest of this guide explains where the line between them falls, and then wires all three together on CDNTR. The demo app in this guide runs live on CDNTR (Node.js, Express and Socket.io): the app, Postgres, the bucket binding, and the presigned upload and download were verified on it on 2026-11-06. The parts marked as options, the Redis adapter, the archive job and ready-made servers, were not part of the demo.

What S3 alone cannot do for a chat

No push. S3 answers requests; it never starts one. When a new message lands, nothing tells the other clients. Without a server of your own they would have to poll: list the room's objects every second or two, from every open tab and phone. That is slow to feel, wasteful in requests, and still not real time.

No append. An object is written whole. There is no "add one line to the end of this file". So each message becomes either its own object (thousands of tiny objects per room per day) or a rewrite of a bigger one, with two senders racing to overwrite each other's version.

No query. S3 can list keys by prefix, and that is all. "The last 50 messages in this room", "messages since I was last here", "unread count", "search for invoice" are all database questions. Keys sorted by time get you part of the way and then stop.

No users or permissions. S3 checks one thing: whether the request is signed by a valid key. Who is in which room, who may see which file, who is blocked: that logic belongs to your app. Handing S3 keys to the browser so it can read and write directly would give every user access to the whole bucket.

None of this is a weakness of S3. It is a file store, and a chat is a real-time, stateful conversation. Use each tool for its own job.

What S3 is good at in a chat app

Attachments. Photos, videos, voice notes, PDFs. They are large, written once and read many times: exactly the workload object storage is made for. The database keeps only a key and a file name per message, so it stays small and fast however much media people share.

Message archive. Old conversations are rarely read but must not disappear. Moving them out of the database into S3 as compressed blocks (one per room per day, for example) keeps the hot table small without losing any history.

Exports and backups. "Download my data", room exports, nightly database dumps: each one is a file, and each one belongs in a bucket.

A lighter app server. Because the browser uploads straight to S3 and downloads straight from it (the next sections show how), the app server never streams media. It signs short-lived URLs and moves small JSON messages, so a single small container goes a long way.

On CDNTR, the S3 service at s3.cdn.com.tr stores objects in our data centers in Türkiye, and it is S3-compatible: the AWS SDKs work with a custom endpoint and path-style addressing.

The architecture on CDNTR, set up with cdnctl

The app runs on Managed Container Apps. The demo is Node.js with Express and Socket.io, deployed from its source folder with cdnctl deploy: the image is built in the cluster, so you need no registry. The app gets an address like https://xxxx.cdn.com.tr, and its WebSocket traffic goes through the CDN (in the demo, Socket.io reported websocket as its transport). If containers are new to you, deploying a container on the CDN covers the basics.

The database is the managed Postgres add-on. Enabled with --env-prefix DATABASE, it gives the app DATABASE_HOST, DATABASE_PORT, DATABASE_NAME, DATABASE_USER and the secret DATABASE_PASSWORD.

The files live in a bucket on object storage. You create the bucket and an access key for it, then bind both to the app with --env-prefix S3. The app gets S3_ENDPOINT (https://s3.cdn.com.tr), S3_BUCKET, S3_REGION (cdn-tr), S3_ACCESS_KEY_ID and the secret S3_SECRET_ACCESS_KEY. The keys exist only in the app's environment, never in the browser.

The whole setup is four steps. Replace <account>, <app>, <bucket> and <key> with the IDs that the earlier commands print.

# 1. deploy the app from source (built in the cluster, no registry)
cdnctl deploy --dir . --account <account> --name chat-demo --healthcheck /health

# 2. managed Postgres -> DATABASE_HOST, _PORT, _NAME, _USER, _PASSWORD
cdnctl container addons enable-postgres --account <account> --app <app> --env-prefix DATABASE

# 3. a bucket and an access key for it
cdnctl object-storage buckets create --account <account> --name chat-demo-files
cdnctl object-storage access-keys create --account <account> --bucket <bucket>

# 4. bind them -> S3_ENDPOINT, S3_BUCKET, S3_REGION, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY
cdnctl object-storage bindings create --account <account> --app <app> \
  --bucket <bucket> --access-key <key> --env-prefix S3

Messages: a Postgres table and a Socket.io handler

A message is a row: room, author, body, and for attachments the object key and the original file name. An index on (room, id) makes the one query a chat runs constantly, the last 50 messages of a room, a short index scan however large the table grows.

The Socket.io side is two events. On join, the client enters a room and receives its history. On message, the server writes the row first, then broadcasts the stored row to everyone in the room, including the sender. Broadcasting what the database returned, rather than what the client sent, gives every client the same id and timestamp, and a message nobody saw saved is a message nobody receives.

socket.data.user is set by your own authentication middleware when the connection opens. The client never gets to say who it is in a message payload.

The demo runs one replica. With several, a message has to reach clients connected to other replicas too; the managed Redis add-on (cdnctl container addons enable-redis) with the Socket.io Redis adapter is the usual way to do that. More on Redis itself in what Redis is.

Schema and message handler (Postgres, socket.io 4)

-- messages table
CREATE TABLE IF NOT EXISTS messages (
  id         bigserial PRIMARY KEY,
  room       text NOT NULL,
  author     text NOT NULL,
  body       text,
  file_key   text,
  file_name  text,
  created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS messages_room_id ON messages (room, id);

// server.js (socket.io 4)
io.on('connection', (socket) => {
  socket.on('join', async (room) => {
    socket.join(room);
    const { rows } = await db.query(
      `SELECT * FROM (SELECT * FROM messages WHERE room = $1
         ORDER BY id DESC LIMIT 50) last50 ORDER BY id`, [room]);
    socket.emit('history', rows);
  });

  socket.on('message', async ({ room, body, fileKey, fileName }) => {
    const { rows } = await db.query(
      `INSERT INTO messages (room, author, body, file_key, file_name)
       VALUES ($1, $2, $3, $4, $5) RETURNING *`,
      [room, socket.data.user, body ?? null, fileKey ?? null, fileName ?? null]);
    io.to(room).emit('message', rows[0]);   // everyone in the room, at once
  });
});

Uploads: presigned URLs, straight from the browser to S3

Sending files through the app server would make it the bottleneck: every photo would be uploaded twice and occupy memory and bandwidth on the way. Instead the app hands out permission and gets out of the way.

1. The browser asks the app for an upload URL, sending the file's type and size. The app checks the user, the room and the size (the demo allows up to 10 MB), picks a key such as rooms/<room>/<uuid>, and returns a short-lived presigned PUT URL, valid for 5 minutes. 2. The browser PUTs the file directly to S3 with that URL. 3. The browser sends the chat message with the object key; the server stores it like any other message. 4. To open a file, a receiver calls the app, which checks access and redirects to a presigned GET URL, also valid for 5 minutes.

A presigned URL is an ordinary S3 request with the signature in the query string, so whoever holds it can do exactly that one operation, on that one key, until it expires. Signed URLs explains the idea in more depth. The app checks the declared size before it signs; keep the limit, the expiry and the key format in the server, where users cannot change them.

Presigned PUT with a signed Content-Type, and the download redirect

import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { randomUUID } from 'node:crypto';

const s3 = new S3Client({
  endpoint: process.env.S3_ENDPOINT,   // https://s3.cdn.com.tr
  region: process.env.S3_REGION,       // cdn-tr
  forcePathStyle: true,
  credentials: {
    accessKeyId: process.env.S3_ACCESS_KEY_ID,
    secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
  },
});
const MAX_BYTES = 10 * 1024 * 1024;

app.post('/api/upload-url', express.json(), async (req, res) => {
  const { room, type, size } = req.body;
  if (!type || !(size > 0) || size > MAX_BYTES) {
    return res.status(400).json({ error: 'max 10 MB' });
  }
  const key = `rooms/${room}/${randomUUID()}`;
  const url = await getSignedUrl(s3,
    new PutObjectCommand({ Bucket: process.env.S3_BUCKET, Key: key, ContentType: type }),
    { expiresIn: 300, signableHeaders: new Set(['content-type']) });
  res.json({ url, key });
});

app.get('/api/files', async (req, res) => {
  // check here that the user may read this room
  const url = await getSignedUrl(s3,
    new GetObjectCommand({ Bucket: process.env.S3_BUCKET, Key: req.query.key }),
    { expiresIn: 300 });
  res.redirect(302, url);
});

The Content-Type gotcha: 403 AccessDenied on a presigned PUT

This one costs people an afternoon, so it gets its own section. Our S3 runs on Ceph RGW, and RGW is stricter than AWS about one header: if the browser sends a Content-Type that is not covered by the signature, the presigned PUT is refused with 403 AccessDenied. AWS ignores that header when it is not signed, so code that works there can fail here with an error that looks like a key problem.

The fix is to sign the type into the URL and then send exactly that type. With AWS SDK v3, pass ContentType in the PutObjectCommand and add signableHeaders: new Set(['content-type']) to getSignedUrl, as the snippet above does. In the browser, set the same Content-Type header on the PUT.

Two details make it reliable. Use one fallback on both sides: a file with no type (file.type is an empty string) must be signed and sent as the same value, for example application/octet-stream. And do not let a library rewrite the header: some upload helpers add ; charset=utf-8 to text files, which no longer matches the signature.

The browser side: the same type the server signed

// browser: ask for a URL, PUT the file to S3, then send the message
const { url, key } = await fetch('/api/upload-url', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ room, type: file.type || 'application/octet-stream', size: file.size }),
}).then((r) => r.json());

const put = await fetch(url, {
  method: 'PUT',
  headers: { 'Content-Type': file.type || 'application/octet-stream' },  // exactly the signed type
  body: file,
});
if (!put.ok) throw new Error(`upload failed: ${put.status}`);

socket.emit('message', { room, fileKey: key, fileName: file.name });

CORS on the bucket

The page runs on your app's address and the upload goes to s3.cdn.com.tr, a different origin. The browser therefore sends a preflight before the PUT and only proceeds if the bucket answers with a CORS rule that allows your app's origin, the PUT method and the Content-Type header. Without it, the upload fails in the browser while the same request works from curl, which is the classic sign of a missing CORS rule.

Bucket CORS rules are set through the S3 API with PutBucketCors. The demo does it once at startup, which keeps the rule next to the code that depends on it. Allow your own origin, not *: the rule is part of what keeps other sites from using your bucket from their pages. What CORS is covers preflights and the common error messages in detail.

A bucket CORS rule for direct uploads and downloads

import { PutBucketCorsCommand } from '@aws-sdk/client-s3';

// once at startup: let the app's own origin PUT and GET directly
await s3.send(new PutBucketCorsCommand({
  Bucket: process.env.S3_BUCKET,
  CORSConfiguration: {
    CORSRules: [{
      AllowedOrigins: ['https://xxxx.cdn.com.tr'],   // your app's address
      AllowedMethods: ['PUT', 'GET'],
      AllowedHeaders: ['content-type'],
      MaxAgeSeconds: 3600,
    }],
  },
}));

WebSocket through the CDN: timeouts and reconnects

The CDN carries WebSocket per delivery rule. On a Managed Container Apps domain WebSocket is on by default, with a 300-second idle timeout unless the delivery rule sets another value; a rule can set anything from 1 to 3,600 seconds. The timeout counts silence, not total length; a connection that keeps exchanging data stays open. Socket.io sends its own heartbeat every 25 seconds by default, which keeps a quiet room well inside the default. The settings are described in the origin connection guide.

Clients must reconnect on their own. Open WebSocket connections can be closed when the edge configuration reloads, and phones drop connections whenever they switch networks. Socket.io reconnects by itself; if you use plain WebSocket, add reconnect with a backoff.

After a reconnect, the client should fetch what it missed. The simplest way is to send join again: the server answers with the room's recent history and the client merges by message id, so nothing is shown twice.

Archiving old messages, or using a ready-made chat server

Archiving. The demo keeps every message in Postgres, which is fine for a long time. When the table grows, a common pattern is a periodic job that takes messages older than, say, 90 days, writes them to S3 as one JSON block per room per day (archive/<room>/2026-08-01.json.gz), and then deletes those rows. "Load older messages" reads the right block from S3 through a presigned GET. The database keeps what people actually scroll through; S3 keeps the rest.

Ready-made servers. If you would rather not write your own chat, Mattermost, Rocket.Chat and Matrix Synapse can store uploaded files on S3-compatible storage. The same three layers apply: their own server process, their own database, and a bucket for files. On CDNTR that means the container, the database add-on and the bucket from this guide, configured with the S3 endpoint, bucket and keys each project's documentation asks for.

Checklist: the mistakes that break chat uploads

CORS rule on the bucket. Your app's origin, PUT and GET, the Content-Type header. Missing it shows up as a failed upload in the browser only.

Signed Content-Type. Sign it into the PUT URL and send exactly that value, with the same fallback for files without a type. Otherwise: 403 AccessDenied.

Short expiry. Minutes, not days. The demo uses 5 minutes for both PUT and GET; a leaked URL is then useless almost at once.

Files do not go through the app. Upload and download directly between the browser and S3. The app signs and redirects; it does not stream.

Auto-reconnect. Expect WebSocket connections to close and reopen. Rejoin the room and merge history by message id.

Size limits. Check the declared size before signing and reject anything above your limit (10 MB in the demo).

Keys stay on the server. S3 access keys live in the app's environment from the binding. Never send them to the browser, never embed them in client code; the browser only ever sees presigned URLs.

Access checks on download. Route file links through the app so it can confirm the user may see that room before it redirects.