Modern Data Stack with Chad Sanderson

The biggest thing that’s affecting us is how the direction of the modern data stack was a pretty clear initial divergence from some of the original data warehousing philosophies. It seems like horseshoe theory is in play where we’re kind of circling around back to a lot of the stuff that was really popular in, like the nineties when it came to data warehouse, design, entity, relationship diagrams, things like that.

Anytime something breaks the data scientists have to go and act like Sherlock Holmes and figure that out or try to throw it to the data engineering team so they can go and figure it out. So, what we’re trying to do is create that clear concept of ownership from the source, having a software engineer, actually own the production code, the events in this case. So we’re using Kafka streams and those Kafka events tied directly to the semantic events that are most relevant to our business, the Semantic events, and the semantic entities.