Designing a solution for searching in high-volume tables

The data-intensive product I worked on had tables with millions to billions of rows. Searching them was slow and resource-intensive. This case study presents a solution to improve the experience of searching in high-volume tables.

The employer was a leading corporate in cybersecurity; I worked as a Senior UX Designer in a team of 20+, fixing UI/UX issues and designing new features on an enterprise web application with a highly flexible, complex interface.

The product was a SIEM (Security Information and Event Management): it centralises security logs from many sources and provides real-time monitoring, detection, and response. Every SIEM has a "Search" page for logs, and that table can hold millions or billions of rows.

Several user types work with the system; this case study focuses on one persona. A typical routine: on receiving a threat notification, the analyst goes to the search page to find the footprints of an intruder.

Users search and filter among billions of rows to reach specific results, using an advanced search tool above the table.

Say an analyst suspects a threat from Estonia and searches for logs originating there.

Traversing a database of billions of rows takes substantial time and resources, so users could wait anywhere from minutes to hours for results.

I started by understanding the limits.

After talking with the project manager and engineering lead, neither was feasible in our circumstances. Every real-world problem comes with constraints.

I was stuck on how to make search faster without upgrading technology or adding resources. Then I noticed something:

To find out, I ran unsupervised observation of user behaviour in the customer's environment with nine users.

Two observations stood out: in 95% of cases users do not find the result in their first search, and on average they run six searches before finding what they need.

That led to the driving insight:

Since users run multiple searches and do not need complete results to decide the next one, my first hypothesis was a time threshold.

Further discussion showed a fixed threshold only covers part of the use cases.

Refining that, my second hypothesis made search real time instead of a fixed threshold.

Was real-time retrieval technically possible? I checked with the Head of Engineering, the PM, a DevOps engineer, and the lead front-end developer. We found limits with non-streaming operators such as ORDER BY, GROUP BY, hash joins, INTERSECT, and SELECT DISTINCT, and designed around them.

I analysed direct and indirect competitors for inspiration on design and approach.

I sketched early ideas, then built high-fidelity mockups and an interactive prototype in Figma (re-branded for the NDA). The prototype let the team compare the current and proposed approaches side by side and feel the difference.

I used the prototype to present the idea, flow, and interactions to the team for validation. After small iterations, the development team implemented it with ongoing support from me, and QA tested functionality and performance.

The impact compounds: if it takes 5 searches to reach a result, each search takes 15 minutes, and the analyst can decide the next criteria from the first minute of results, the time saved is dramatic.

We then tested with end users:

A total of 180 searches were recorded across the two interfaces.

The top three challenges, then a closer look at how I handled one.

To understand the technical side and design a robust solution, I voluntarily took a Splunk course on how cybersecurity specialists search logs to uncover evidence.

The biggest lesson was the gap between the stated request and the real need.