ThoughtSpot and dbt Labs partner for semantic layer integration

To further strengthen our commitment to providing industry-leading coverage of data technology, VentureBeat is excited to welcome Andrew Brust and as a regular contributor. Watch for his articles in the Data Pipeline.

Business intelligence (BI) specialist ThoughtSpot and data modeling darling dbt Labs are announcing a new partnership that brings together philosophies and technology. More than just a business tie-up, the two are integrating their platforms, such that models built in dbt can immediately service the semantics of search expressions articulated in ThoughtSpot. The idea is to harness the work of analytics engineers to facilitate self-service analysis by business users.

The vision

VentureBeat spoke with ThoughtSpot co-founder and CTO Amit Prakash and dbt Labs CEO and founder Tristan Handy. Prakash said that for ThoughtSpot: “[In the] last few years there’s [been] a lot of focus on the developer persona and the data engineer and the analytics engineer persona, which led to us…opening our back end with ThoughtSpot Modeling Language and a lot of APIs…and in that world, dbt became kind of natural partner to work with, because dbt seems to be the most popular tool of choice for analytics engineers.”

Dbt Labs’ Handy seems just as jazzed about the alliance: “…the thing that I’m excited about with the ThoughtSpot partnership is that…companies spend thousands and thousands of human hours encoding the knowledge of their organization into dbt and then they need a way to actually do something with it…make a decision, or flow it into a downstream process. And so this vision matches very, very nicely with ThoughtSpot’s vision of last mile analytics…getting it in front of every business user.”

The implementation

The integration brings dbt transformation and semantic models into the ThoughtSpot platform. Using the dbt integration wizard, users can import a dbt model from within ThoughtSpot’s data workspace, as shown in the figure below.

Event

Intelligent Security Summit On-Demand

Learn the critical role of AI & ML in cybersecurity and industry specific case studies. Watch on-demand sessions today.

Watch Here

The dbt integration wizard in ThoughtSpots data workspace

Once the dbt model is imported, ThoughtSpot’s platform can instantly detect table relationships and metrics defined in it, and make them available to search. And in the data workspace itself, ThoughtSpot worksheets are automatically generated, complete with column descriptions from the dbt model displayed right after the column name, as shown below.

ThoughtSpot worksheets reflect metadata and semantics contained in a dbt model, including column descriptions

Referring to dbt’s transformation product, ThoughtSpot’s Prakash summed the value up this way: “there’s a lot of rich metadata in there that we were asking users to enter again, and in the way that ThoughtSpot uses it, that integration makes it seamless.”

This lets companies model their data centrally, while self-service users get the same semantic context they’d get if all that modeling work were done from scratch in ThoughtSpot itself. The benefits surface in both the mainstream ThoughtSpot user interface as well in application-based embedded analytics experiences implemented via ThoughtSpot Everywhere. Customers can even generate scriptable ThoughtSpot Modeling Language representations of dbt models.

Analytics market alignment

The BI world of 15 or 20 years ago was based on a few monolithic enterprise stacks. While that didn’t make for a lot of choice for customers, it did mean the components within each stack were fairly integrated and worked together well. Meanwhile, the revolution in the analytics industry of the last decade or so has brought about a disaggregation of those stacks into separate commercial and open source components. That’s great for innovation but it has made for a lot of duplicated efforts and a lack centralized infrastructure.

As a former data practitioner, who was at the same time very business-focused, dbt Labs’ Handy wanted his company to focus on bringing order to this big collection of technologies, which he calls the modern data stack: “My data background is that I am hybrid of some technical and a lot of business. And all of this modern data stack tech, starting with Redshift and Looker and Fivetran and then eventually Snowflake and BigQuery…all of it seemed to be built for me, and so I wanted to build a company that was focused on figuring out the best practices for how to use this stuff.”

Handy sees the effort as a success: “We’re trying to build this standardized…infrastructure layer across the modern data stack…and we’ve kind of become the standard for how knowledge is formulated inside of the modern data stack.”

Playing well together

But that’s not enough, because data-driven business is about more than getting technology components to talk to each other; it’s also about getting people in different roles to work together, which seems to be what the ThoughtSpot/dbt combo is addressing.

On that point, Handy commented: “Our persona is the analytics engineer…[a] hybrid, technical and business person who understands how to encode business knowledge into the tables and views that are in the database. But those analytics engineers…no company ever has enough of them to answer every business question. You really want the line-of-business people to be able to answer their own business questions, and that’s where ThoughtSpot comes in.”

Prakash commented reciprocally: “to fully specify a[n analytics] question, you need a lot of information…some of that information is going to come from the business user and some of that information is going to come from the expert or the analyst or analytics engineer, who’s defined the metrics, who’s defined the joins, who’s figured out which columns need to be addressing which questions…and we need good ways to capture that. If you look at ThoughtSpot’s product itself, there is a pretty rich semantic layer but it it’s not a reusable thing…and so the power of having this in a semantic layer in dbt is that this becomes reusable. And I think there’s benefits to a company just purely focusing on semantic layer and expanding the vision there.”

Scatter/gather

Ultimately, the modern data stack is also a fragmented one. In the old days, things were too concentrated and innovation was lacking. But now things are too dispersed and siloed. Handy analogized this situation to “AOL versus the Internet” and rightly observed that “it took us a long time as an open Internet to recreate some of the things that AOL delivered in 1994.”

Sage words, as it’s taking the modern data stack a long time to deliver some of the things that the enterprise BI vendors delivered in the mid-2000s. Some of that has to be reconstituted and dbt’s approach, especially when partnered with modern BI platforms like ThoughtSpot’s, starts to put the industry on that path.

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings.

Source