Supporting a World-class Documentation Technique with Atlan
The Lively Metadata Pioneers sequence options Atlan clients who’ve accomplished a radical analysis of the Lively Metadata Administration market. Paying ahead what you’ve discovered to the following knowledge chief is the true spirit of the Atlan group! In order that they’re right here to share their hard-earned perspective on an evolving market, what makes up their fashionable knowledge stack, revolutionary use circumstances for metadata, and extra.
On this installment of the sequence, we meet Tina Wang, Analytics Engineering Supervisor at Tala, a digital monetary companies platform with eight million clients, named to Forbes’ FinTech 50 record for eight consecutive years. She shares their two-year journey with Atlan, and the way their sturdy tradition of documentation helps their migration to a brand new, state-of-the-art knowledge platform.
This interview has been edited for brevity and readability.
May you inform us a bit about your self, your background, and what drew you to Knowledge & Analytics?
From the start, I’ve been very thinking about enterprise, economics, and knowledge, and that’s why I selected to double main in Economics and Statistics at UCLA. I’ve been within the knowledge house ever since. My skilled background has been in start-ups, and in previous expertise, I’ve at all times been the primary particular person on the info crew, which incorporates organising all of the infrastructure, constructing stories, discovering insights, and many communication with individuals. At Tala, I get to work with a crew to design and construct new knowledge infrastructure. I discover that work tremendous attention-grabbing and funky, and that’s why I’ve stayed on this area.
Would you thoughts describing Tala, and the way your knowledge crew helps the group?
Tala is a FinTech firm. At Tala, we all know in the present day’s monetary infrastructure doesn’t work for many of the world’s inhabitants. We’re making use of superior know-how and human creativity to unravel what legacy establishments can’t or received’t, so as to unleash the financial energy of the International Majority.
The Analytics Engineering crew serves as a layer between back-end engineering groups and varied Enterprise Analysts. We construct infrastructure, we clear up knowledge, we arrange duties, and we be sure knowledge is straightforward to seek out and prepared for use. We’re right here to ensure knowledge is clear, dependable, and reusable, so analysts on groups like Advertising and Operations can give attention to evaluation and producing insights.
What does your knowledge stack appear to be?
We primarily use dbt to develop our infrastructure, Snowflake to curate, and Looker to visualise. It’s been nice that Atlan connects to all three, and helps our strategy of documenting YAML information from dbt and mechanically syncing them to Snowflake and Looker. We actually like that automation, the place the Analytics Engineering crew doesn’t want to enter Atlan to replace info, it simply flows by means of from dbt and our enterprise customers can use Atlan instantly as their knowledge dictionary.
May you describe your journey with Atlan, to date? Who’s getting worth from utilizing it?
We’ve been with Atlan for greater than two years, and I consider we had been certainly one of your earlier customers. It’s been very, very useful.
We began to construct a Presentation Layer (PL) with dbt one 12 months in the past, and beforehand to that, we used Atlan to doc all our outdated infrastructure manually. Earlier than, documentation was inconsistent between groups and it was typically difficult to chase down what a desk or column meant.
Now, as we’re constructing this PL, our objective is to doc each single column and desk that’s uncovered to the tip person, and Atlan has been fairly helpful for us. It’s very straightforward to doc, and really simple for the enterprise customers. They’ll go to Atlan and seek for a desk or a column, they will even seek for the outline, saying one thing like, “Give me all of the columns which have individuals info.”
For the Analytics Engineering crew, we’re usually the curator for that documentation. Once we construct tables, we sync with the service house owners who created the DB to grasp the schema, and after we construct columns we arrange them in a reader-friendly method and put it right into a dbt YAML file, which flows into Atlan. We additionally go into Atlan and add in Readmes, in the event that they’re wanted.
Enterprise customers don’t use dbt, and Atlan is the one manner for them to entry Snowflake documentation. They’ll go into Atlan and seek for a selected desk or column, can learn the documentation, and may discover out who the proprietor is. They’ll additionally go to the lineage web page to see how one desk is expounded to a different desk and what are the codes that generate the desk. The very best factor about lineage is it’s totally automated. It has been very useful in knowledge exploration when somebody isn’t aware of a brand new knowledge supply.
What’s subsequent for you and your crew? Something you’re enthusiastic about constructing?
We’ve been trying into the dbt semantic layer up to now 12 months. It’ll assist additional centralize enterprise metric definitions and keep away from duplicated definitions amongst varied evaluation groups within the firm. After we largely end our presentation layer, we are going to construct the dbt semantic layer on high of the presentation layer to make reporting and visualizations extra seamless.
Do you’ve got any recommendation to share along with your friends from this expertise?
Doc. Undoubtedly doc.
In certainly one of my earlier jobs, there was zero documentation on their database, however their database was very small. As the primary rent, I used to be a robust advocate for documentation, so I went in to doc the entire thing, however that would stay in a Google spreadsheet, which isn’t actually sustainable for bigger organizations with thousands and thousands of tables.
Coming to Tala, I discovered there was a lot knowledge, it was difficult to navigate. That’s why we began the documentation course of earlier than we constructed the brand new infrastructure. We documented our outdated infrastructure for a 12 months, which was not wasted time as a result of as we’re constructing the brand new infrastructure, it’s straightforward for us to refer again to the outdated documentation.
So, I actually emphasize documentation. Once you begin is the time and the place to essentially centralize your information, so every time somebody leaves, the information stays, and it’s a lot simpler for brand spanking new individuals to onboard. No person has to play guessing video games. It’s centralized, and there’s no query.
Typically completely different groups have completely different definitions for related phrases. And even in these circumstances, we’ll use the SQL to doc so we are able to say “That is the method that derives this definition of Revenue.”
You need to depart little or no room for misinterpretation. That’s actually what I’d like to emphasise.
The rest you’d wish to share?
I nonetheless have the spreadsheet from two years in the past after I appeared for documentation instruments. I did loads of market analysis, taking a look at 20 completely different distributors and each instrument I may discover. What was essential to me was discovering a platform that would connect with all of the instruments I used to be already utilizing, which had been dbt, Snowflake, and Looker, and that had a robust assist crew. I knew that after we first onboarded, we might have questions, and we might be organising loads of permissions and knowledge connections, and {that a} sturdy assist crew can be very useful.
I remembered after we first labored with the crew, all people that I interacted with from Atlan was tremendous useful and really beneficiant with their time. Now, we’re just about operating by ourselves, and I’m at all times proud that I discovered and selected Atlan.
Picture by Priscilla Du Preez 🇨🇦 on Unsplash
