Quickwit takes on Elasticsearch with an open source search engine for large datasets

Join today’s leading executives online at the Data Summit live now! Watch here.


Let the OSS Enterprise newsletter guide your open source journey! Sign up here.

Search plays a fundamental role in just about every modern application, from Amazon and Netflix, to Slack and Salesforce. On top of that, every application generates swathes of log data, which includes timestamped information about events from inside the software — this might be details on what resources were accessed, the runtime characteristics of an application, and anything relevant to the operation of that system.

Being able to search and make sense of all these computer-generated logs is important, because it helps companies troubleshoot errors and bugs, solve bottlenecks and latency issues, comply with regulations or internal security policies, and better understand what’s going on under the hood. It all constitutes part of what is known as “observability” — that is, the ability to measure the internal state of a software system by analyzing the raw outputs.

Typically, companies pool all their log data in a centralized database system such as ClickHouse. But administering and managing all this comes with many challenges, while the costs associated with storing log data can lead companies to ditch portions of it. This is a problem that Quickwit is setting out to solve, with an open source, cloud-native search and analytics engine built for large datasets.

Founded in 2020, Quickwit touts its ability to run sub-second queries on terabytes of data on object storage services such as Amazon S3 — and it promises to do so at up to “ten times cheaper” than Elasticsearch. But Quickwit isn’t necessarily designed as a direct replacement for every scenario covered by search incumbents such as Elasticsearch — it has more limited, targeted use-cases in mind.

To build on the momentum it has built since its first release last July, the company today announced a $2.6 million seed round of funding co-led by FirstMark and firstminute, with participation from a slew of notable angel backers including MongoDB cofounder Eliot Horowitz; Dataiku CEO and CTO Florian Douetteau and Clément Stenac; and SendGrid founder Isaac Saldana.

‘Cost-efficient’

Paul Masurel: Quickwit cofounder and CEO, and Tantivy creator
Paul Masurel: Quickwit cofounder and CEO, and Tantivy creator

Built on top of the open source, Rust-based search engine library Tantivy, which was created by Quickwit CEO Paul Masurel five years ago, Quickwit is a distributed search engine that serves up additional features on top of Tantivy’s “low-level building blocks” for search. Quickwit delivers a REST API to index data, perform search queries, manage indexes, and manage clusters, with a suite of pre-built connectors spanning data sources such as Apache Kafka, Amazon Kinesis, and Amazon S3.

At its core, Quickwit is aimed at so-called “immutable” datasets (data that is never deleted or updated), which makes it perfect for companies wishing to store and search log data. It also promises sub-second latency of as low as 140 milliseconds, which is just fine for log management searches. However, it’s well understood that milliseconds matter in the online world, which is why Quickwit isn’t targeting use-cases such as ecommerce websites which require lower latency.

“A search query requires at least two round-trips to object storage — however, object storage systems have higher latencies than local disks used by systems such as Elasticsearch,” QuickWit cofounder François Massot told VentureBeat. “The bottom line is, Quickwit can never answer faster than 130-140 ms, which is acceptable for the use-cases that we target, but not for ecommerce ones where higher latencies correlate with losses in sales.”

Quickwit cofounder Francois Massot
Quickwit cofounder Francois Massot

That all said, Quickwit does compete with the likes of Elasticsearch for some purposes, including searching through logs, cloud storage backups, and providing full-text search functionality for online analytical processing (OLAP) databases like ClickHouse. But it’s in these use-cases that Quickwit hopes to differentiate itself enough to win the hearts and minds of small companies and enterprises alike.

For starters, object storage is cheaper to store data than hard disks, while Quickwit is written in Rust which is known to consume less memory than Java, which is what Elasticsearch is based on. Moreover, Quickwit’s decision to separate compute and storage may set it apart in the analytics space — it claims to be the first open source search engine with such an architecture in place.

“Quickwit instances are stateless and can be started or shut down in seconds — you don’t have to move data like in Elasticsearch, as storage is separated from compute,” Quickwit cofounder Adrien Guillo explained.

Quickwit cofounder Adrien Guillo

In theory, this all translates into faster and cheaper for Quickwit’s target use-cases. And given this cost-efficiency promise, companies may be more inclined to retain more log data which will improve the insights they garner into their system’s performance.

“A lot of companies end up reducing their log retention in order to curb their cost,” Massot added. “Quickwit makes it unnecessary to throw away this valuable data.”

In terms of the types of business Quickwit is gunning for, Guillo argues that it will be suited to companies of all sizes. Smaller firms will want to use Quickwit as their log search observability building block, while bigger enterprises will build entire applications on top of Quickwit, spanning application and log management, search analytics, data lake search and analytics, and more.

“Companies struggle to run existing search systems at scale, and must mobilize considerable resources and capital to do so, especially for ever-growing number of logs generated by applications, systems, and business events,” Guillo said. “Quickwit offers an unparalleled cost-efficiency.”

While Quickwit didn’t give much away in terms of its early customer base, Guillo did confirm that they worked with French unicorn Contentsquare on a proof-of-concept.

For now, Quickwit’s business model is based on a simple dual license approach — an open source AGPL license for free use, and a commercial license which includes support and gives the licensee a “voice in our roadmap,” Guillo said.

While the company may consider offering a SaaS in the future, this isn’t currently on Quickwit’s agenda.

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn More

Source

Follow me on Twitter:

Leave a Comment