So we're going to talk about 4 main areas. The first area. We're going to talk about is data. Lake kind of key concepts and then we'll go into the design patterns will talk about governance and will talk about security so the first question, we need to ask is why so is anybody ever heard this phrase. Data is the new currency is that a thing over here.
Kind of interesting because if I have, like a dollar bill right, I can put it in my wallet. And that's a good place for it. If I have a stack of money, I might need to use a safe and if I have AA big bag of money. I need to put it into the bank and the reason for that is because while the wall. It's appropriate for a dollar bill it's not appropriate for a big bag of money.
Going to scale right and so, if we think about treating our data as an enterprise asset, and if data is the new currency. We need to take a same approach to managing our data is as we do to managing our money. So a data. Lake really gives us that it gives us that scalability. It gives us a framework to manage the data in a lot of this were already kind of doing that.
With our SSIS packages and archiving are incoming files archiving data after runs and loading it. So that the data. Lake really does is it gives us solid framework for managing that So what are the key characteristics that we're trying to get out of a data Lake so it needs to be interoperable it needs to work with all the different.
Source is so massive parallel processing things like Azure data. Bricks, which is based on spark things like Impala hive SQL. DW it needs to be interoperable. That's the first thing because if we put our money in and we can't get it back out. That's a problem right. It needs to be performant. So it needs to be linear scalable as we add more data too.
Keep the same performance characteristics right, it needs to have infinite scale. I don't want to have to call it and say, I'm out of space in the data. Lake can you add another 10 gigabytes right or 10 petabytes so I need to be able to scale on demand and I only want to build for what I'm using it needs to support different access patterns right so analytical workloads searching.
Specific pieces of data and needs to be very flexible from that standpoint needs to be secure. It needs to support role based access control. I want to be able to use groups and principles to provide only the access to the areas that they need to see so it all makes sense and then it needs to be manageable so ideally want to scriptable API, preferably rest and if you think about it are opera.
Or, a protocol like web HDF sore HDFC? What are we going to put in the data. Lake we're not gonna put money in the data Lake but we are because data is the new currency. So we have different types of data that we got to look at so when you think about the Big 4 VS of big data. We think about volume variety.
Velocity so on, and so forth so if I think about the data. This is really the variety in the 4 VS I have structured data now when I say structured data. All data has structure doesn't matter if it's a word document or a movie file or database table, but specifically in this context. When we say structural we're talking about Relational.
When I think about semi structured those are things like our flat file formats. So there's a format there. There's a structure but it's unstructured 'cause. It's not coming from a relational source and then when I think about completely unstructured data. I'm thinking about things like images or opaque binary formats, so if I think about how could I implement a data Lake.
Just could I use I could use Azure Blob Storage. So it's an object store. It has a flat structure. There's no file note folders. There's containers and there's files. It's optimized for unstructured so it's optional eyes for things like ved images. Video there's some other objects that it supports but it's primarily optimized for unstructured data.
Or just much cheaper than a DLS so that it can be a good thing and it steered I can set up the redundancy that I want I can set up the performance level do? I want a cool tear do? I want hot here. So I can set up different levels of performance. So if I think about that. That's not really ideal for my data Lake right there.
That are compelling, but some that are not so if I think about Azure data. Lake Gen one. Where do I have? I have interoperability? It works with all the major MPP flat forms. I have performance? Is optimized for analytic workloads. Azure Blob Storage. I believe has a file limit somewhere around 2. Terabytes maybe a little bit to the South of that.
Because they like storage has no file size limit if I think about security with blob storage. I don't have the Posix Ackles. I don't have active directory. I think authentication. What I have is I have secured access tokens. Things of that nature. I have manageability have a folder structure right So what?
We want to take the best things of Windows Azure Blob storage and take the best things of Azure data. Lake storage and have those characteristics in my data Lake and that leads us to Azure Data Lake Gen 2, which was recently reduced so that has the deep integration that we're looking for has hierarchical namespace, so folder structures and it gives us atomic file.
Your data Lake Gen, one if I do a metadata operation. Let's say. I need to move files rename am so on. And so forth each of those operations is going to be a delete and recreate here. I've got atomic file operations. It's much more efficient flexible pricing. I now have tearing which I didn't have an ADL S version. One there is a little bit of pricing premium over.
But you're getting a lot of capabilities, there and I have improved performance. Just all up so if I think about implementing the data. Lake generally what I'm going to do is I'm going to want to go with Azure Data Lake Gen 2. If I'm looking to maybe back that data Lake up. There is not a great story for that right now, So what I would do is generally I would use a tool.
We are this DP from Hadoop and copy that data to Blob Storage, which is cheaper and so that would be the reason that I would I would do that, so that's something to think about. There is not a great backup and recovery storage right now, so file formats. I've got a lot of different file formats that I can choose from. So we'll walk through a few of those text file it's human read.
And many custom formats. It's well supported by the tooling today. JSON fairly common these days. It's very interoperable. It supports complex nesting which can be useful. It's human readable. But the drawback is file sizes can get very large. It's very verbose so if you don't use compression, it can be law.
Afros are really good file format to use because it supports schema evolution. It's actually a file that has 2 components as a binary component where the data stored and it has a JSON base schema and so you can do things like if I have a row in there. That doesn't map. The current schema. It can map defaults. So it supports a very flexible schema. However, the one of the downfalls.
Some custom viewer utility to browse Avril files or see an another file format. We want to talk about parquet are what's known as columnar formats from the big data world. So RC came from Horton works in Facebook. It's a columnar format. It supports supports internal compression as well as external compression generally speak.
Analytical workloads, I'm going to want to use either or see or parquet. So parquets an alternate competing. Spec from Cloud Erin Twitter. Its columnar format supports compression. It supports predicate pushdown. The drawback to using these files because of that internal compression and because a lot more complex internal structures slower right.
So if I'm if I'm optimizing for writing, I want very efficient writing, I may use text or JSON. Perhaps Avril probably ever if I'm optimizing for analytic workloads. We're going to run multiple queries against that file. I want to use either parquet or perhaps orc questions so far.
OK next thing I need to think about when I designed my data. Lake is zones right so all zones are. It's a fancy term for folders, so one of the key things that they like successes proper organization of your folder structure. You're going to have some common zones archive raw reference cure.
Stage and we're going to talk about these a little bit later in the design pattern section. But the key concept here is we're going to sort the data by the purpose. We could create other zones. So some people say, well. How would I implement a dev test and prod scenario with Data Lake. So I could create data. Lake instances for each of those environments. That's probably the best approach.
Also, just create zones in there for dev test and prod and restrict those with Ackles Write Access control list. I would probably have subzones inside of each of those zones for subsystems. So let's say. I'm going to bring in 2 source system so I'm going to bring a worldwide importers and adventure works. I'm going to create a sub zone under RA for worldwide in importers. I'm gonna create.
Under all for adventure works I may create subzones for users as well in the sandbox area. Those are examples of subzones is that all makes sense. So some people called use different terminology for those zones. So I'm actually going to define what those zones are in a few seconds. I do want to talk a little bit about how we're going to access that data so this is going to be.
The engine so we're going to use something like Azure data brakes SQL. DW those are probably the 2 main ones. Today there's some other ones like Impala kudu so on, and so forth. Some older technologies. I'm going to refer to his legacy. It just means that they're not the 1st place. I would start off with if I was doing a new project so first would be a.
Azure Data Lake Analytics, which uses a language called you sequel, which I actually really like the language. But Microsoft chose to kind of move everybody over today to break so they're not going to do a lot more continue development with you sequel. That's the future of that and then high of course, right so high. It's based on map reduce its not's fastest.
Mark we're not going to get into the reasons why, but generally speaking, if I'm looking for a big data processing platform. I'm going to land on Azure Azure. Data bricks or SQL data warehouse is that makes sense. Anybody use anything else for MPP processing.
OK.
And you don't need to use MP as well. So I could use just standard SQL, DB right or whatever. Relational engine that I want. That's an option as well. So let's get into some of the data Lake design patterns.
File format so we're going to fundamentally store the data in files and we've got to think about some things So what I like to do is I like to include the source or the linneage of the file. So where did it come from? What does the data represent and I also like to include the time so I know when the file was generated and I want to make that unambiguous inconsistent.
File into another place. I want to know exactly where it came from so it might seem a little redundant. I have all this information in here, but there's a reason for it, so in this example. This data is coming from a sales order table and I've got the year. I've got the month. I've got the date and I've got the hour that the data file was generated and it's a CSV file.
Does that make sense and then you know you could do things like put different underscores in different places to make the time parsing a little clearer. You know you could kind of play with that a little bit. I could certainly add the schema name in front of the file. I could I could have the source name. But the point. There is just do something that's consistent unambiguous to support it.
OK.
So another thing I have to think about is partitioning and partitioning is related to a concept called split ability. If I have. Let's say have a very large 2 terabyte sales order table if I if I'm going to process that it's not superefficient have it in one file because I can have multiple compute nodes working against it.
What I might want to do is split that file up by date so in this case I may use the update date of when the transaction occured or the create date. Sometimes there's a little bit of conflict there because from an efficiency standpoint. I want to write the data down by update date so that I only can doing append only operations.
Analytics standpoint, I usually want to do it by create date right so I have to kind of resolve that and figure out and the zones can help me with that because in the raw zone where I'm landing data. I can use update date. But when I put the data into a curated zone. I can use the create date does that make sense.
OK and I want to go to the grain that I need to support the processing so one of the things that data. Lake doesn't like is hundreds and thousands of files in the same directory. So if if I'm finding myself usually by default what I'll do is I'll go down to the year month and day level. That's usually sufficient. But if I'm getting files every minute or.
I'm a break that out by by hour or minute as well. I don't want to so if I don't have any activity for a day. I don't want to create a bunch of empty folders right? That's just going to add to maintenance overhead and I'd certainly don't want to create a bunch of empty files or mixing different file types within that folder hierarchy because the way most empty systems work is.
To the folder level and it processes all the files within now there's ways. You can get around that with wild card, so on, and so forth, but this just keeps everything nice and tidy So what is split ability mean with ability means that if I have this 2 terabyte sales order file?
MPP engine can actually split it into segments so not all file formats are splittable that's why we need to think about what's the correct file format to use for the workload. Also, if I compress a splittable file format. It may render itun splittable because of the compression algorithm. So you need to think about that and using an unsupportable file format isn't.
It means that if I do that. I have to do more work on partitioning. Then I have to start breaking underneath that folder structure. I start breaking that out into multiple parts. But it can be very cumbersome to manage. It is what I want to do is I want to balance it. So it's the right size. So that we keep all the different empty. Nodes busy when they're processing the file from a reading and writing standpoint.
I need to think about is compress ability so like we said before compression can render a format on Splittable and also keep in mind that some of the columnar formats like Oregon parquet have internal compression that you can then layer on top with other compressions, like gzip LZ oh snap EETC so test.
German compression makes sense So what are you really doing when you're using compression?
You're just basically trading right so we have 4 different resources in a computing environment. We have storage data, which is IO Network and then CPU so whenever I compress the data. I'm reducing the storage and the disk. IO in the network bandwidth. But I'm trading off CPU so it's going to increase the.
And sometimes that's the right decision to make sometimes it's not right. So you just need to weigh that off.
Um so generally speaking for analytical workloads. You want to use columnar formats. Why? Because they're more expensive to write which isn't necessarily a pro. But if I need to query. That same file multiple times for different work or I'm going to do predicate pushdown or I'm going to use a subset of the columns, so column elimination.
Columnar formats a good choice does anybody can anybody think of Technology in SQL server proper that's very similar to orcher parquet.
Column store index. NG yeah, so parquet an orc are very similar to column store in indexing. They both use things like run length. Encoding dictionary stores and all these things to make those deficiencies right so it's very similar technology. So, whenever you hear Oregon parquet just think clustered column store index and then that leads you to make the right decision do I want?
Performance or do I want a clustered column store index for analytical performance makes it very simple right.
Let's see so let's talk about data Lake zones again right so we're going to go through some of the zones. So my raw zone. That's where my data lands so other people might name that inbound right ingest what have you. The key thing. The key rule of this zone? Is it's immutable. I'm never going to update or delete data from there.
Tend to it now that's very dogmatic right? That's not realistic. We realize that data inserts updates and Elites. So how do we resolve that we'll get to that in a second? There's no transformation. I'm getting and keeping the data in its incoming state now. Obviously, if it's coming from a message queue like Kafka or event.
But I may need to serialize it to Jason but generally. I want to keep it as close as possible to the original state? What that allows me to do is if somebody questions, a result, in my analytical layer. I can trace it back to that original data and I'm never going to delete it for the most part right. So I can always go back to this was the data you sent me so this output.
That's the linneage right? That's veracity of the 4 VS. I want to organize my raw zone into subzones sorted by source system. And if it's a structured data like coming from an Oracle or SQL server have a folder for schema folder for each table and then if the tables large enough, I can split it into your month date, I can partition it further.
This obviously like I just I guess started say earlier. It's assumes we're doing an incremental load process right. We're not going back and deleting the data so in a little bit will talk about a pattern to reconcile that so the curated zone is the zone where I want my end users to run from this is where my analytic zones.
The data is enriched it's been transformed it's very similar to my data warehouse so if I think of my RA is my staging area. Or maybe my ODS my curated zone is really my data warehouse or my data. Mart right and depending on my use cases in requirements. I may have lots of different folders in there to support things like data.
Cox outbound things that are used for integration datamart type areas. The data is blessed. So it's well governed and it's a good place to let the end users go in and do self service analytics from.
Questions.
So we talked about handling the inserts are sorry the updates and Elites. So the mutable aspect. If I need if I have data. That's coming in, where I'm not getting that incremental load, then what I'm going to do is I've got generally 2 patterns. That kind of happened in the real world, so the first is I'm going to get multiple intraday loads.
Maybe there are full snapshot of the data set from the source system or a partial data set with updates and Elites. So I would I would do is I'd land those into temp and then I would further down in my pipeline. I would compare temp to my ra zone and I will work out what's the current state of the data based on that and then I would write.
Intera that solves that problem now, I've got, I've got to have a process in place to manage the movement of the files and generate that but that's how you're going to deal with mutable data in the raw zone. So another thing that might happen is I may get for an analytical work load. It's not efficient operate off thousands and thousands of.
In data from a service bus is coming in every minute or I'm getting very small message based transactions. I may use temp is an accumulator or I'm going to land. All those files down in their in their raw format using no columnar format just using JSON or text or what have you maybe even XML and then maybe every half hour? I'm going to write a vacuum process?
It goes on gets all those files combines it and dumps it out to Park. A or or one of those files so that I'm kind of I'm doing kind of like a micro batch. I'm batching that data together for analytical purposes.
And that talks about one of the anti patterns, which is the small files pattern, which will talk about it in a little bit archive zone is another critical zone to think about so it might be the case that from an analytical standpoint the business only wants to work with 5 years of data but from a compliance standpoint, they may need to keep the data.
10 years in that case for an efficiency standpoint. I'm going to move the data. That's outside of that window. I'm going to move it into the archive zone and that could just be the last stop before we actually completely either. Remove it from the data Lake or move it to cheaper storage such as blob storage. One of the interesting features of ADL SV 2 is each contain.
What you have is you have a storage account that supports multiple containers each container in that storage account can have its own tier. So it can be optimized for activity, which is the hot tear or it could be a cool tear which is similar to like an S 3 glacier where you're paying a little bit less, but your responsiveness is less, I can also restrict the access to.
Through the different Taco list or access control entry, then what I would use for raw or sorry for curated so I can keep people out of the archives own if I don't want them to see that, but give certain people, the right to access that data if they need it for compliance standpoint, so that's zones. Now you may have some other zones that you would do some common one.
Be like sandbox where I'm just going to, if I'm just exploring a new data source. I just need to land it in there. It's kind of like the Wild West. Anything can go in there and it's just for playing around with you may create a zone for each user that needs access to there so they can play with their things staging is another one, so if I'm going to load data into SQL DW or into another.
Make Rita area that's just for staging I could also re purpose temp for that. So does that make sense so this is all the things to take away from this session. This is probably the most important thing is to use good zones and good file naming to organize your data link yes Sir.
Oh, it within the data like it really just depends so ADF could certainly also the question was how would I handle moving the data between zones and ADF, so Azure data. Factory could do that. It's very good for orchestration. But it could also be something like SQL, DW supports create external table.
A lot of times that's kind of almost an anti pattern of moving data. There's some cases where you definitely need to do it more commonly you're just generating a new data set which is kind of moving the data. You're transforming it and then moving, it so in that case. I could use something very simple like data breaks any of them PM.
I could use other workflow schedulers like uzi. I could use air flow. So there's a lot of different technologies. It kind of depends on which fits into your stack, but generally. I'm going to use Azure Data factory. But the interesting thing about it doesn't actually support a move operation. You have to do it through the HTTP APIs the rest API.
Cumbersome but you can do it, you could also use AZ copy. There's a lot different tools that you could orchestrate with ADF, so don't think about Azure Data factory in terms of being a data movement tool. It's good for moving data from source to sink. Not so much moving data around like what we could do with a file system task.
It's actually a really good question.
Any other questions.
OK, so now let's talk about just some higher level, architectural patterns as they relate to data ingest so I've kind of simplified this. I usually go over a couple. Others, like Lambda Kappa. But today, we're just going to talk about batching streaming so batches kind of this notion that as time moves on I take data and I process it and.
The high level diagram of what in architecture might look like and what the services. You might use in an Azure framework. So very similar to Alex' S question what would I use. Obviously here orchestration. I could use data factory. Uzi I'm doing batch processing. So I've got a data Lake inside a data sync and I'm doing batch processing with one of the tools there.
Front that with an analytical store and sometimes even an engine like Azure Analysis Services or power, BI for analytics and reporting streaming's a little bit more interesting case with streaming what I'm doing is I'm processing the data in near real time. Now, some streaming systems are actually micro batch so if I think of spark structured streaming it.
A true streaming environment would be something like storm right so storm actually continuously runs and processes. The data as soon as it arrives. The options there. Usually, you're using either Kafka or event hub and you're using stream analytics or perhaps mark streaming or storm and there's generally less orca.
Because it's always running right same operational stores. Same story there, So what are some of the anti patterns and challenges that we run into so the biggest one we run into his data swamped the scenarios there. I'm mixing file types within a folder. I don't have any folder organization.
One catchall folder, I have no standard file naming I'm not using any kind of data governance or catalog. The big problem with Swamp is. It prevents self service, which is the whole point, so the whole architectural concept of data Lake gets to have all my enterprise information in one spot right so it's a data hub and then I'm pushy.
For a specific use cases 2 engines that are optimized for that, so today. Lakes very good at providing cheap infinitely scalable storage and it can power. MPP engines to access the data. But there's no compute inside of their small file problem. When I have lots of small file problems that not only slows down my empty engine.
But it adds a lot of different overhead and so as I mentioned earlier, the best thing to do is accumulate. Those anusa batching or micro batching approach to bundle those up into larger chunks and an important thing to note is the optimal file size depends on the file type as well as the compute engine so different.
For different block sizes right usually you generally speaking, 6464, Megs is a good starting .2, 56 MB. But if you think about like a terabyte data source. That's a lot of files still so you even have to bring that up so you want to multiple of one of those usually and then each of the compute engines will have specific tune.
Going to play with different settings to set for your specific data. Lake storage right so challenges so there's a lot of challenges. I don't expect you guys to go through and read. All these but it's a lot of the things that I've been talking about throughout the day right so one of the things that we haven't touched on that's a good lead up to the next thing we're going to talk about which is Gov.
If we don't have any data cataloging or metadata. It becomes very hard to go in there and find the data set that I want so the next part of this session is going to really hit on that.
Questions so far, about design patterns anti patterns or challenge is how many people here, actually using a data Lake right now.
So the rest of you are looking at using it.
So let's have a few minutes. Let's talk about some kind of common misconceptions. So do I only need a data? Lake. If I'm using IOT data?
How many people say yes?
OK, so you guys know and then it's certainly OK to have a data Lake that doesn't necessarily mean that you have to have like, Hadoop or spark or one of these things right, I can use a regular SQL database right.
So a couple of things we need to think about from a governance perspective. Our data cataloging does everybody understand what I mean? When I say data cataloging has anybody heard that term so a data catalog is just a metadata repository where I can register data sources, so if I think about just a common enterprise scenario.
ERP system my CRM system and when I say register. I'm just storing the connection information in there and then I'm going in and cataloging the tables. So just like information schema. Orsis tables inside of inside of SQL server. I'm bringing that metadata out of that source so that it's browsable when it's searchable and it's discoverable. I'm documenting it.
And I'm at M enriching that metadata.
So let's talk about data calico so I'm going to show you guys very brief walk through of Azure Data catalog. It's not seen a lot of further extension. So it's a perfect example of what we call MVP or minimally viable product, but it does get us started down the route.
And what I would recommend if you don't have a catalog spin up Azure Data catalog. It's very cheap right. It doesn't do a lot of stuff that you're going to want to do. But as you go through and use it and you hit those limitations. You'll have a checklist of things when you go out and evaluate other vendors that you might want to move to and if you really find stuff that you're stuck you can give them proud.
And see if maybe start taking it up again, but the key thing that it does is it allows registration documentation and search and discovery and then we're going to do a quick demo. But first I want to show some kind of common metadata columns, so this is data that I would put inside of maybe JSON files or.
X files or even add these columns to my parquet files. So when was the file created most of the time I can get that from the file creation date stamp attributes but sometimes inside of a MPP engine. It's not as easy to surface that so I'll actually add a column that says the create date and that's the create date of the file process data.
Access data ingested, it, I may also store the MD 5 hash of that original source file and that's really important because from a linear standpoint if I'm tracing down and I have some data that the business users telling me is wrong. I want to go back and make sure that nobody altered that file after I wrote it down to raw right and the best way to the cheapest way to do that is using them.
And then I want to document the source so what's the Lenny edge of the data. What source system did it come from this becomes really critical when you're working in a multi instance environment so let's say you're a manufacturing company and you have multiple factories that are all running the same ERP instances of it at each factory or?
Have a manufacturing engineering system, where it's getting factory data. I wanted to say that oh. This data set came from the Mexico factory or this data, fact factory came from the factory in Frankfurt right so I want to know where it came from so I can track the data back because it may turn out that I have a problem with my TL feed so I want to be able to.
Bad data is coming from or maybe there's a process breakdown right. So it helps me trace back and do continuous improvement.
Questions.
Um.
So another part that we're going to talk about which I had slides but I figured we could go through and do some demo of it is just stuff around security like? How do I set up security to access? How many people here? Are you actually using Azure?
So most of you that's good, so things like setting up users groups principles to actually access the data. Lake and then different areas, where we set the access control entries and talking through that there's some common problems. I see when I go to clients where they struggle a bit so with that, we're going to take a quick tour.
And see what we have.
So the first thing I'm going to show you guys. I think I'm going to start with data catalog so I've already opened that up, I can create a data catalog from the Azure portal. One thing to note is it's a single tenant resource, which means that in a tenant if I think about Azure. I have a tenant and attendant can have multiple subscribe.
So if Alex creates a data catalog in my tenant and I try to create one. It won't let me so it's a global resource and a lot of times people in Azure. They get really see new things, they say. I want to create one and see what it does and then they forget about it. So you may have to go back and delete a catalog or you may have to take ownership of it, you can only have one and So what this is showing me.
Option to publish a data source. I'm not going to go through that workflow but you can see. I've actually added some enrichments here, so I did a search here and I found my reference zone right and so I can add a friendly name. I can put a description here, I can preview a data set now for the textual data format.
Don't provide preview support so that's a major gap in my opinion. But if this was a sequel table. I could actually import a preview of that data and see what it looks like. Here's where I could get the columns now again if it's a SQL data. Source is going to automatically determine the columns for me, but because this could be a parquet file or it could be a JSON file.
Not enough to determine that metadata so I could go in here and add columns right so if it was a sales order header file. I could add my column give it a data type and I can add descriptions here and what's really nice about documenting this stuff is I can actually there's a good rest API, so that I can go in and actually pull this data out so I can.
Great documentation, I can do generation of ETL assets or tables for further downstream. So it's a great repository for metadata that I can use for multiple purposes and then it gives me this documentation thing and they were so close on this rich text so I can do some formatting, but if they would have.
It would actually have been a lot better right because I could I could do a lot more things with it, but anyway, it's a good place to say you know document. The purpose of the zone. So you notice here that I've actually imported the zone as a resource into the data catalog so if I go in let's see in search for sales.
With.
You can see here I've got uh.
Sales order.
And that's actually pointing to probably zoom that a little bit. That's actually pointing to a parquet file so one of the things if I think if you think back to what I said about file naming you're not going to register each and every file within the folder right so for sales order. I really only need to register at once. So it one of the tricks that I do is.
Processing systems if a file's name starts with an underscore it won't process. It'll ignore it. So you can create a dummy file so that you can still register it and that way. I've got the location 'cause if you see here I can say explore data. Lake and will actually pop it open one of the things I didn't talk about that. It actually is super useful here.
Our tag so I can actually tag it and do some lineages there and I can search on those tags right, so very similar in Azure to tagging and then connection information. So this is really cool. It not only gives me the URL where the file is but I could put some text in here say if I want to get access to this data set I can put in a helpdesk ticket with like service now or something.
I can do all that and I can build automation around that workflow.
So that's kind of useful.
So that's the data Lake and that's really the start of our data governance right. It gives us some metadata. It gives us some documentation gives us some workflow any questions, yeah.
So the question is, is how how would this handle schema evolution is that we were thinking. So if the file format evolves and I've got a document. It you actually hit a pretty big gap because there's no way to really version that column attribute information right so it's really just this is the most recent snapshot now.
Because the one good thing that they did about this is even though they made some very minimal decisions. They they didn't enforce kind of approach so you could handle that by for example, in that documentation section. You could just try to version one which is from this date to this date has this schema version 2, so you could have.
And there's nothing to stop you from hotlinking that to another source. It's not ideal right. So I'll give you that, but there's there's ways that you could at least call it out. It's this game. Evolution is kind of a big problem. Even even Avro doesn't completely solve in my opinion, I mean, it gives you a good step because at least a query that runs.
Specs this column and it's not there for the older data, it won't just blow up but it really just kind of kicks. The can down further to the processing because even if I can get that into my staging area right and that's 1 of the big things about data. Lake is schema and read right? That's the big, big reason for us to do that, then do use another persistent staging layer technology like.
Kim Bor in one where they bring it into a relational table problem with that is as you guys know if you weren't in SSIS package on top of that and the tables schema breaks. Everything falls apart right schema and read. Just says Alright. Whatever you want to throw at me and then you put the scheme on it. When you're ready to starts consuming it. So it really gives you 2 really big benefits on top of.
Having to do all that ETL development front just to get the data loaded to do some analysis on top of it.
It also gives you the ability to start collecting the data right now because if it takes me. Let's say 3 weeks to start doing ETL, but that data is getting overwritten on the source system. I'm losing all that history so I want to collect that data very quickly so I hope that kind of gives you some things to chew on any other questions so.
Left I'm going to look at some of the security stuff. Just very briefly. One thing I do want to show you guys as well.
Um is storage Explorer so his storage Explorer is a Microsoft product that allows me to browse my different data. Lake stores and while that's going up just 'cause we're running so short on time.
I'm actually going to go into it through another way, so this is the Azure portal. And I'm going to just go into data. Explorer, which a web based browsing experience. One thing to be aware of this experience. If I open a file. There's some files a little preview, but it has a limit so it will look like the files corrupt because it cuts it off because of the character limit but if I actually die.
Files there so just be aware of that you can see here, I've got some of my zone set up.
Right so those should look familiar. I've got an archive of God curated. I've got raw. I've got references a new one that's just for reference. Datasets I've got my temp and you can kind of see. I'm going to go through the curated something that's going to be most interested so I just created his own called DW within that I've created by Dems all my DMS are going to live under there.
Facts underneath here right so it's just keeps everything organized now that's typically if you think about curated zones are very use case driven this would be the case where I'm going to build my entire data warehouse in there, so I would have other zones on top of that. That would define specific customer use cases so if there's a data science that needs to map customer churn for example.
I might have a customer churn zone where I put that curated data set in there, but then I also have this for self service. So I could stick power BI on this and go from it right so let me go through real quickly. Ackles so this is a key thing and I want to call out them. Des Moines ADL SV, one, not because it's the best.
Because it's the easier to demo because I've got this browser based experience, which I don't have with the blob store 80. Gen 2 right now, so if I go into add here. One of the things that you need to understand is I can set up a user so let's see. I'm going to pick a DF access.
This is a setting where people go wrong, so again. I'm going to want to use best practices. So I'm going to service principles in groups, but let's just say for the sake of argument. I want my ETL process. That's my ADF access account to have read write execute So what I've got to do is I've gotta set my mask, which is going to be 755 so read right.
The Unix file permissions, I'm going to say this folder and all children. So it's going to apply that recursively down the chain and then I need to check this third axis. This is a key one where people usually mess it up. It's not enough to say a default permission entry. It needs to be an access permission entry that allows me to access the file and then uh.
File so if somebody if I write it. It's going to set that 755 to the file by default for new files. Oftentimes, what happens. If somebody creates a new folder and doesn't get that default underneath it. And so they write a file and then nobody else can read it. So usually if you just set those generally like that now for specific scenarios, you May.
Very little bit more, but you gotta have that another last thing to note on security. It's not like windows, so I can't just say I you have read from this file down. I need to have execute all the way to the folder and then read from there down to allow somebody to read so it's a little bit different than windows permissions that's really.
So what I have to show for the security stuff just going back to storage Explorer real quick so.
Let's see if that's going to go. I don't care about that, so again the experience is very similar right. I'm just browsing files. The actual management is all here. I can do folder statistics to see how big the file is. I want to show you guys, just real briefly as well. Before I wrap up so this is Gen 2.
Notice it calls it out as Gentoo preview even though it's GA right now and here's where I can actually create containers so containers are the things that I can make hotter cold, So what I could do is I could create my archive zone in the cold container and create the rest of my zones in here. Once I go in here, though I now have access to create my file folders right so I've gotta.
First and then create the hierarchical namespace within that container right and so the next step is going to be kind of merging some of the old blob storage. APIs right now. You can use one or the other with Gen 2. They're going to start merging those APIs as the product evolves, so that's about all I have for today, if there's any questions, we can.
I hope it was useful so and hope everybody had a good sequel bits.