Thanks for coming everybody. Welcome to AA very, very quick brief introduction to data flows. This is a gigantic topic will be looking at things from an introductory POV first of all. Let me introduce myself, my name is Chris. I'm a consultant and trainer specializing in power Vin analysis.
So as such, I'm very, very interested in data flows it's the cool new thing. Here's all my contact details. Follow Me Bluff. All of those things now as I said, I'm a consultant and trainer and one of the questions that people often ask me when I'm on doing training or consultancy is they say Chris what's the app.
Way to learn a product what's the best way to go in depth about a particular feature. You know should I buy a book? Should I watch a video should I like experiment on my own do you wanna know the secret the secret is? When a feature like data flows comes along you find a conference about 3 or 4 more?
You submit a session on it, even though you don't know anything about it and I guarantee you will spend the last 2 weeks before the conference learning in absolute detail in complete panic. Everything about data flows 'cause you can read the documentation. You can watch the videos and of course, everybody Alberto said.
Things look very, very easy you only really find out how it works when you decide that you're going to do a session on it, you're going to be standing up in front of 100 or 150 people and really know what you're talking about the other problem with data flows is that they are a preview feature now some preview features in power BI.
Basically work but they put the preview tag on them just to kind of cover themselves data flows are a proper preview feature. As such stuff might go wrong. There is stuff that happens that I'm not quite sure why. It happens and it happens, inconsistently so I have lots and lots of backup things which.
Just in case stuff goes wrong, I tried to show data flows on my pre con on Wednesday and guess who want power. VI was down that afternoon, so I couldn't even get into do anything. Hopefully will be lucky it today. OK, the next 50 minutes. We will be going through looking at what data flows are how you create.
Why you should use them and then look at some more advanced topics like linked entities computed entities the AI features in data flows the common data model all kinds of things like that. Some of this stuff is brand. You there's at least one feature that I've got permission to show you which is actually released, but it hasn't been an?
Only store it by accident yesterday when I was testing. My demos and I've got permission to show it, so there is some guaranteed cool new stuff here that you have never seen before.
Right to start off let's ask the basic question what are data flows?
First thing data flows of the latest example of some guy in Microsoft marketing, making a very, very bad decision because of course. We've got data flows all over the place, the data flows. I'm talking about are nothing to do with Azure Data Factory. They are nothing to do with Microsoft flow, which is of course, also part of the power platform.
Or, a power BI feature.
Put simply data flows are a self-service ETL feature, which means that you can split your ETL away from your data set right now in the world of power by you. Open up power. BI desktop you open the query editor you import some data.
That you model, it, you do it all in one place.
Data flows allow you to split the ETL part away from the dataflow so effectively. It's kind of like power query as a separate service running in the browser. Still, running inside power BI, but checking out data, which can then be reused between multiple.
So it's a way of sharing raw data between multiple datasets. It is not a way of sharing pieces of a dataset between multiple datasets. We're not talking about sharing dax. We're not sure talking about sharing calculations or relationships or anything.
All we're doing is we're saying let's run a dataflow. Let's go grab a table of data that table of data will be shareable between multiple different sources were going to see lots of examples of this to explain in more detail.
Open word about the licensing data flows are not available to anybody with the power be. I free subscription. As a result you will not find them in your my workspace workspace. They are only available in workspaces? Which of course, only available to people who've got a power BI pro subscript?
Slide power BI pro now I'm not sure and I don't. I don't like to be cynical. I don't like to be cynical about Microsoft all the time, but
This is not necessarily a way of Microsoft trying to screw more money out of you in the background power. BI premium is going to very different architecture compared to power VI pro and there are some things, which are only really feasible or easy to implement in power. BI premium so the things that I'm going to show you today, which a power BI premium. I don't know.
Will always be power BI premium they've just implemented them in power. BI premium first but will see is that in power, BI premium. There are going to be, it will give you the ability to handle larger data volumes. There will be certain new features. You'll get better. Refresh performance and you get. These AI features that I'm going to be talking about.
All right.
Concepts what is a dataflow?
Data flows exist within workspaces.
You create a dataflow within a workspace, but a dataflow actually is really just a container data flows on their own don't do anything a dataflow is a container within the dataflow. You have multiple entities and an entity is the interesting thing an entity is one of the.
Later that we're going to be able to share between multiple datasets. Why didn't. They just call it a table. I don't know, but then I think there may be a good reason. But anyway, data flows contain entities and from the power VI. POV at least an entity is going to be.
All the power query query and then expression.
So what does this mean?
Traditionally, if you think about developing power BI right now. You've got a data source. Let's say this. The database with a table. In it, you've got a workspace and you've got 2 datasets inside it. Let's say that both of these datasets need.
Same table.
Right now in the world of power by what happens is that you build your 2 datasets. You publish them You refresh them.
And when table data set a gets refreshed the table goes from the data source router table to data set a? When they set be gets refreshed. The table goes to data set be so we're hitting the source system twice everything is completely.
With datasets sorry with data flows we've got an intermediate step in the middle.
So what we're able to do is we're able to call our store system once.
The table go to the data flow. It will turn into an entity and when we refresh. Data set a in data set be we get the data from the entity in the data flow and we do not go back to the source system so it is literally just the kind of staging area where.
And we can hold data.
Now.
So, in order to turn on data flows we're going to have to go to the power VI portal. We are going to have to go to the admin portal.
And for a start we need to go to our.
Tenant settings scroll right down to the bottom and there is a create and use data flows option. It's in preview. It's not on by default, so you want to have this turned on if you're using a premium capacity.
You're going to need to go to your capacity settings. This takes a few moments to turn up for my demos here, I'm using?
And a theory skew power BI embedded if you ever do want to go away and test power. BI premium features buying power. BI premium is obviously incredibly expensive. Instead, you can go away, and get a power BI embedded capacity. It's still expensive, but you can turn it on and off so you only pay for the stuff that you actually use.
Power BI embedded has all of the same features available as power VI premium. I've got a power BI embedded a force. Q here, which is basically the same as AP one for premium now. If you were using premium as an extra step that you have to go through.
Nice will take a little bit of time again data flows are a work load on a capacity so within a power. BI premium or a power BI embedded capacity. There are features that take up resource is and you can have you can turn those features on off and you could specify what resource?
Cool.
And you can see that data flows are.
A work load here that I've turned on and I've said they can use a maximum of 30% of my memory on there. You can see that I've got paginated reports. SSRS reports available. That's turned off, I've got to turn on my AI. This is what I'm going to demonstrate later this datasets preview only appear the other day, I have.
Yes, but I'm sure it will be announced soon.
Another thing that you may optionally want to turn on is this here. What I've got turned on here is the ability to use some external storage for my data flow. If I want I will explain what that is a bit later, but there it is there's another thing here.
Another thing to mention about building in using data flows is that all of the functionality of a data flow is only really available in a gnu style workspace so as you probably know we are currently in transition between all that workspaces and new app workspaces and if you.
8 A knew app workspace you will see this preview option. Now, if you click try. Now, this will be how you create a new app workspace and if you want to use data flows. I strongly suggest that you use knew app workspaces. 'cause all the cool stuff is only properly available.
Workspace so let's call this sequel bits.
And I'm storing this on my premium capacity. I'm not going to store the data for this in my external storage so we click save there.
OK so that's how we can figure a dataflow, creating data flows is pretty pretty easy if you are used to using power query. So power query up till now has been a kind of on premises thing. It's been embedded inside power be. I desktop it's been embedded inside Excel with the power.
But power query like everything is going to the cloud. So what we've now got is something called power query online, which is available inside the power VI portal. It's something that's also being used inside Microsoft flow. So we'll see the online version of power query in various different places, but
Create a data flow into create entities you create your workspace you created dataflow and then you open up power query online and you use it and more or less in the same way as you use power query on premises like I said this is where we will meet the buggy?
Slow uncertain functionality you can use power query online through the why and it will do pretty much everything you want. There is some data sources missing right now, which will appear in the future. There's some functionality missing in the why that will probably appear in.
It may be a better idea for you to build your queries that you want to create as data flows inside the power query editor inside power. BI desktop and then just copy and paste. The M code into power query online because you've got an advanced editor window there. There's no intellisense. It's very old school but you.
Nice job and in most cases, everything will just work now we need to build a dataflow and this is where the second source of stress. For this particular session came for me because of course.
Whenever you're doing a demo for power BI. You've got to have some cool interesting fun data. You can't use some data source that everybody's already seen before. You've got to do something interesting and off the wall. So I literally spend about half a day earlier this week. You said II leave all my preparation to the last minute spent half a day earlier this week.
Where on earth? Am I going to find a new API where I'm going to find a new data set so I sitting thinking well, what can what can I do that hasn't been seen before? What does SQL bits need SQL bits? Is an almost perfect event? But what does SQL bits need SQL bits needs more beauty it needs more arts and he's my?
It is a bit of a culture free zone isn't it really, and as it happens.
I found a rather cool new API to play with for doing demos, you only Dutch people in the house.
Oh, that's lucky that's lucky.
Have you all been now excuse my pronunciation?
We will be into the The Rights Museum is that is that an awful mangling of the name.
Alright so OK, so the rights museum in Amsterdam is obviously one of the great museums of the world.
It also as a very enlightened approach to open data and APIs.
So what we can do is we can visit the Rights Museum. I've logged in. I'm putting everything in English and they've got this feature called right studio way you can explore.
And you can look at pictures.
And if we look at a particular picture.
You've got some interesting object data about that picture now alot of museums like to keep hold of their IP rights. Museum says no. Let's make it all open data. Let's let everybody get hold of all of the all of the interesting information about this and image.
And let's let people get hold of this through our API, which you can sign up for you just get a key it works really well so this is what I'm going to be doing my demo with so let's say we've been to the Wright Museum. We've got a feature here where we can create our own little.
We're interested in so this is my first collection or something. I love particular about this great translation. This is not the collected works of Chris Webb. I didn't actually take these pictures.
I wish I had but here we got a collection of some of the masterpieces that I found here. And I thought OK. Let's do something with this we've got a whole series of interesting pictures.
And if we take a picture like this.
We can look at the object data each picture has got its own object number.
SKC 1672 and if I've signed up to the API. I can get hold of the information for this picture in Jason format through a simple cool to a web service and there, I've got that information that I need.
So if we take this and we want to consume this through.
A dataflow I've got my empty workspace here. I'm just going to skip through to the main screen data flows are turned on. I'm going to create a new data flow by going to create data flow.
I'm going to define a new entity.
Her role of the data sources supported out of the box from the why like I said, not everything. That's available in power, BI on Prem power query on premises available here, but web APIs are.
I'm going to paste the URL and here I am going to go through my own local Dick Gateway. It's got a key in there don't steal my key but click OK.
Click next keep your fingers crossed that this works.
And here we are in power query online and we can do all the same things that we can do on premises. So let's drill down. Let's find this label information. Let's convert this to a table.
And let's call this was this lady's name. Alida something will call it that I've built my query. It's a nice simple query. Let's call make this text.
And I've built my entity so one query becomes one entity. I've gone through done everything in the way that is very, very familiar to anybody who's used power query on premises. You can see. I've got basically all of the same options up here that I would have in the ribbon in the power query editor plus.
Cool.
Click done.
We've created our entity, we got one entity inside our dataflow. I'm going to click save.
Let's call this pic.
We're saving the entity the next thing we need to refresh. Sewing to say. Let's refresh. This now, hopefully this will be nice and quick.
And we've created our entity. Now the next thing is, we need to consume it. If I open up an empty power. BI desktop file. I want to consume. The entity that I've just created in power, BI desktop remember this is just going to be available as a table? It's
These tools just like any other it's just this is a new data source that lives in size path side power. VII can go to get data. There is a new power. BI data flows data source, which Microsoft think is going to be so popular. It's available here on the.
Drop down menu for popular data sources.
I'm going to go to power BI data sources.
I can see under here all of my workspaces.
Inside, a workspace I can see my data flows an inside my data flow. I will see my entity.
And there, we are. It's refreshed it's available as a table now when I load this.
By the way I can edit this so I can do whatever I want here in the power query editor, but when I load this.
It is going to go and get this data from the data flow. It is not going to call the API anymore. It is going to go and get the data from the data flow.
So that we are there's our table of data?
So we've created and configured and consumed a very, very simple data flow.
Just out of interest. This is a very simple example. What I've got pre prepared over here, something that I'm going to use in my later demos is a slightly more complicated query. I've got an Excel worksheet here with the whole table of my interesting object numbers and that collection that I showed you.
I have got.
A function here we look in here, which is going to allow me to call the API for each one of those object numbers and the end result is going to be this. I'm going to get a table with a list of object numbers in titles descriptions and URLs to some nails.
People if we have a look up here at my data flows demo.
I've got a dataflow here called art.
And if we look at the entity here. Let's edit entities you can see that I've got my function available. Just like in power be. I desktop I've got my query here. Luckily, it runs. This is where I was getting some weird authentication issues, but there's my big table and I'm obliged.
Being kind of cool and funky looking so let's get data from this table.
Which is going to be cooled?
Art.
Click edit I can clean up things here because it just works just like any other data source. Let's call this title.
Let's call this.
Description.
Let's call list.
Thumbnail.
Will close and load edit this loaded in?
This should be nice and quick because it's just being read from the dataflow. Remember we're not hitting the API anymore. Let's build a table actually before we do that. Let's go to our table here. Let's set a thumbnail to be of data category image URL were obliged to do something With you know nice.
I'm not generally good at nice pictures, but we put title and description thumbnail an if the Internet is working nicely. Let's also make the grid a little bit bigger.
Hopefully now at least are Dutch speaking friends in the audience will be a little bit better educated about their culture and.
Everything so that all of my Dutch descriptions and a little thumbnail. Let's make the thumbnail bit bigger as well so you can see it.
We are, we built our.
Rights museum reports so we can learn a little bit more about all of these pictures.
That we are OK, so we managed to cool our API, you get the data out.
This leads us on to first important question. What are they for OK we've seen the technology but the real question is? Why do I want to use data flows the first important thing to say about a dataflow is that it is not a replacement for a data warehouse no matter who tells you this some sales guy comes up and say.
We don't use the data warehouse data flows of the cool new thing. They are not a replacement for a data warehouse. A properly structured properly modeled properly managed data warehouse is still a good thing and a dataflow doesn't change that. But as we all know there are going to be situations where you don't have a date.
There are going to be situations where the data. You need is not yet in the data warehouse and may never be in the data warehouse. I find it unlikely to believe that information from the rights museum in Amsterdam is ever going to be in my data warehouse. But I might still need to be able to build a power BI report from it so.
Data warehouse a dataflow can help you get some but not all of the benefits of a data warehouse So what are the benefits that a dataflow will actually give you?
As we've seen when you create a dataflow you create a staging area to hold the output of your data and if you've gotten multiple datasets that consuming these tables of data, it can be quite expensive to be able to have to go back to an unstructured orb.
Data source to keep getting the data. You know if data set a refreshes we go back to the source system and that takes 30 minutes, then data set be refreshes. It goes back to the same school system does the same thing and that takes 30 minutes? Well, you know you've lost 60 minutes of time overall.
So by taking the data out of the data source. Paying the price to hit that slow data source. Once you pay that price once and then when you get the data from the entity stored in the data flow that should be much, much faster so you're going to get a performance benefit as well. Those of you have done a lot of.
Power query will also know that even when you just run a power query query. Once it has a strange habit of hitting the data source more than once, and if your data sources slow that can really slow things down, so staging the data inside an entity in a dataflow can help get over that because getting the data from the data flow will be.
And if you're hitting it false data source rather than something slow that can speed things up as well.
Also, if we going back to the source system and we are refreshing multiple times. Getting the same data multiple times. This can cause problems for the source system. You've got an ERP system and somebody's going away and sucking all of the data out of it several times a day. This can lock up the store. The source system and cause problems.
To be actually doing if you hitting a web service that can not only be slow there might be a rate limit there as well, telling you OK, you can only extract this many rows of data a date and I've run into that several times myself, so by taking that initial load extracting one staging it, you get around those types of problems.
Obviously, if you've got multiple users building datasets, grabbing different chunks of data from different places. There is always going to be a risk that somebody makes a different decision. Somewhere they grab the same data. But they filter it. There apply different business logic in different ways, which means you end up with inconsistent reports.
The dataflow gives you health try and get around that, if as in this case that I'm doing the demo here, you end up having to do some quite complex stuff to get the data out of the data source. It means that you can share a complex query between multiple datasets and then let people do their own quite simple clean up after all, and then.
It will be in a situation where you need to share some data that has no data source. It's quite common for example, that you want to be able to share something like a date dimension table for your company through a dataflow that might be something you generate in M script with no data signals at all, and then a dataflow is another great way of being.
The output of that M query, so everybody inside. You organisations so there are lots. And lots of benefits data flows even if they're not a data warehouse.
That covers everything that you can do with data flows if you're a pro customer who's got premium.
A surprising number of people so with premium you've got some extra features. The 2 key features that you get with premium are linked entities and computed entities now like I said with.
A dataflow a dataflow lives inside of workspace.
An entity lives inside a dataflow, but some users may only access via have access to some workspaces So what you can do with linked entities is you can make a dataflow that lives in one workspace an entity inside that dataflow that lives inside one workspace appear.
Quite a dataflow in another workspace, it doesn't duplicate the data. It is literally just a link to the data flow in the original workspace, so it allows you to create the dataflow once and then just make it magically appear in various different workspaces, so that everybody can see all of these different workspaces.
So if we go back to our.
You work space that we just created earlier. Maybe I want to create a linked entity that art.
Workspace sewing so that our data flow in the other workspace. I'm going to go to dataflow here because this is on a premium capacity. There's my little diamond. I'm going to create a link Tensity instead, and create a link tensity. I just go through this little wizard.
I can see what I've got here, basically my workspace is my data flows.
And there is my entity that looks all good click next and you can see here that this entity has a little link on it, I've got a warning.
I can't make any changes here to this linked entity. I can make it available if I click done.
The entity will behave like any other but I can't add any other steps inside the Link 10 stick. It just as a way of saying here's this entity. It's available inside this workspace as well.
So no data duplication going on.
To be clear with a linked entity you create your entity in one data flow in one workspace. It becomes a linked entity and then you can. Refresh to all of the different datasets in all the different workspaces. This is where I can show my first never at well my only bit of functionality. There's not ever being seen.
Um.
This was something that was going to be on my road map, but it's available today, so we go back to my.
Workspace with all of my data flows demos then.
Yes. Let's save that as well.
Let's do that.
Will refresh that.
Now let's go back to here you can see that I've got a lot of data flows with a lot of entities in it. When I was preparing this. This is my list. View I've now got a diagram view.
That will allow me to see all of my entities and all of my links my data sources and the problem with this is that if I zoom in so I can actually see it. It's all still pretty small and there is no way of popping this out to full screen. But I can see how the data flows from all of the different data sources.
Toward the link testis that's quite nice but still get the preview feature.
The Second Main.
Premium feature is computed entities computed entities are basically entities inside your data flow that use other entities inside the data flow is a data source now, what's happening here is that when your first entity runs the output of that entity is going to be staged physically inside.
When you create a computer density. It works, a little bit like referencing inside power query. Except that that referencing inside normal power query. You are not staging any data in any intermediate point with the computer density. You are with a computer density? What you're doing is you're saying.
To assist the output of the first entity and then use the output of that entity persisted somewhere as the input to another entity, which is also available inside the dataflow and this again can have some important performance benefits. So to be clear what's happening is that we take our table, we create?
We can create a linked entity and then use the output as a table inside our data set if I go back to my original sequel bits.
Workspace here we go to data flows and go to art.
Like I said, I can't create knew things in my linked entity here. But if I right click, and reference this.
We call this computed this is a computer density its input will be the staged persisted output of the art entity and you can see that it's got this little Harry Potter lightning sign and anything. I do here will not hit the original data source again.
So that's got some important performance.
Improvements.
In terms of refresh. There's not all that much to say you can. Refresh the manually you can. Refresh them on a schedule like any other data source. If you're using premium. You've got the ability to use incremental refresh. But it works like incremental. Refresh for regular datasets? What I find is that people often say call. I just add the new data on to my.
Well, not really incremental refreshes designed with more complex data warehousing type scenarios in mind. Hopefully, they'll make a little bit easier.
If you're using linked entities there's a little bit of a gotcha. If you've got a link tensity inside the same workspace as your original source entity when the original source entity gets refreshed the linked entity will get refreshed automatically but if you're going to link 10 city in a separate workspace.
Be refreshed when the original soul sensitive gets refreshed which could I think catch a lot of people out.
Right let's get on to the fun stuff right at the very end.
One of the things that is possible with the architecture of a premium capacity is as I said, this idea of work loads of new features that could only really exist in the cloud and of course, Microsoft being Microsoft having splurge loads of money on AI and machine learning needs to prove that it's going somewhere with it and we've got some.
Coming in today to flows as well. AI features that will allow us to do cool things inside data flows to prep our data. There are in fact, 3 things that were going to be able to do with AI features in data flows. We're going to be able to call some of the cognitive services. APIs we're going to be able to train.
Classify simple machine learning models. We also going to be able to call Azure machine. Learning models that we've created in Azure machine learning to proper data as well. I'm only going to show the cognitive services stuff because I'm not a machine learning person and I'm not going to embarrass myself.
So.
If we go back to here let's close this.
I've got a dataflow with some translated text. I could actually do this demo on the Dutch text, but I did think that perhaps that might be.
A little bit of skewer for most of the audience.
So let's create a new data flow.
Come on.
Close this.
So let's create a new dataflow here, I'm going to create a gnu linked entity.
And if we go to data flows demos. I've got a text. Analytics data flow here where I've got the translated text for each one of those works of arts, so let's go to English text here.
That's the translated description, I didn't translate this myself.
Let's create a gnu computed entity here will call it.
Analytics and you'll see that I've got an AI insights button up here. This is where I'm going to be able to do power query. AI type transformations, so let's take this.
This is where things might end up being a little bit slow, Unfortunately, but we want to. For example, detect language invoke this. This should be nice and fast.
Because of course, all of this text is.
See waits he waits in English.
Works in a very, very power query type way, so we detected the language. Let's make that.
Text.
And now let's add something a little bit more interesting. Let's for example, do some sentiment analysis again on the text column. I've already got the ISO code here for English, so I can take that.
And now I've got a sentiment score for each one of these pieces of text. I can even.
We go into here extract key phrases.
From here.
No, that's that should be that click invoke.
And what I will get back in a moment's This is one of those situations where computed entities are useful because we absolutely don't want to be going back to the right museum. API calling getting all of this data, multiple times. And then sending it in to the cognitive services. API and you can see here it's not necessary.
But I've got a list of.
Key phrases extracted from the text of all of these different things.
As another demo that I'm going to not be life, but I will show you very quickly. I've got an Azure. Blob storage store with all of the Jpegs for those images downloaded an if we go to hear let's cancel this.
I can actually do some image recognition on it.
Camera.
Go to image tagging.
And what I'm doing here is connecting to my blob store getting all of the binaries.
And in a moment, we'll get back is a list of all of the things that have been found in the images. We go back up to here and have a look at my pictures. I'll show you the output of what it's found for each one of these 'cause I've got it pre stored.
Let's go back to here.
Let's go to get data power BI data flows.
Data flows demos image tagging.
This is going to show us things a little bit more easily.
And if we click edit.
Here we can see for example, that it's found in picture 1983, 15, Glass Container Cup ceramic Ware porcelain.
And it's pretty clear that it's found that in that image, which I think is pretty cool.
Some things it doesn't do too well. Some things it does surprisingly well, even for pictures there's Vincent and I think he is.
If we go back down to here wearing hat and old.
But this one picture is bird outdoor.
Now, has done a pretty good job, I think.
OK, so we can do some fun AI stuff, which is always good for demos.
Just finish up, I've said that all of this stuff is persisted somewhere. It's persisted as CSV files stored in Azure Data Lake Storage. Gen 2, which is a optimized storage service created by Microsoft for BI and analytics type.
If you're pro user you gotta limit if 10 gigabytes of storage available out of the box for you as a user if you're using premium you've got 100 terabytes of capacity for workspace, which is going to be quite a lot. If you need more or you need more flexibility. You can actually buy your own Azure storage.
You can link that with power BI and tell power be. I to say for this particular workspace create all of the data flows and store the output. Of these data flows in my own Azure storage. The dive, Bolt now. This not only allows you to work with larger amounts of data.
This is where we talk about the last thing which is common data model integration so the format that a dataflow outputs it date. Its data in is not some obscur undocumented undocumented internal Microsoft thing it is the common data models.
Light output now the common data model is something that Microsoft has been pushing quite a lot. It is a lot of marketing stuff a lot of stuff you're probably not going to be interested in. It's all tide up with this thing called the open data initiative, which is a way of SH.
Tweens different systems and different vendors so initiative created by ASAP Adobe and Microsoft. The common data model is a standard and extensible collection of schemas. That represents the business concepts. Basically, some developers had the idea of saying instead of all of those database people building their own data.
When we just create one database schema that everybody can use now is that a good idea. Yeah, well will think about that. I've never seen one size fits all approach is ever work. But there is some useful stuff about this because it specifies a particular storage formats, so everything so.
Model formats is stored as a series of CSV files plus a Jason metadata file which is an open schema here is a Jason metadata file here. Not too difficult to generate and if we have a look in my Azure storage here this.
Put.
Of.
A very simple.
Data flow that I created there's the Jason file and I've got a folder here called simple sales.
That has the CSV files CSV file snapshots of all of the output. So I can actually see where this data is being persisted.
Why is this interesting?
Well, it's interesting because not only power BI data flows can create data in this format. The idea is other applications and services by Microsoft by third parties written by you will also be able to create common data model type folders in.
Storage and then you will be able to attach these as data flows inside power. BI so for example, in the future. Azure Data Factory Azure SQL DW Databricks. All of these things I guess. We'll be able to output data into common data model folders.
You go to your workspace.
You create cancel this.
You create a new data flow.
You attach an external CD M folder. I'm going to cheat here this is actually.
The URL for the folder from the other data source. I'm not sure where this is meant to work.
And to the folder path here create and attach.
And now you can see I've got an external attached data source. The contents of this so long as it conforms. The standard could be generated by anything but it's available and Consumable as a data source inside power. BI so you could re writing your own application to output data storage is CSV file and it will be available at the data.
To everybody.
Last thing just close up.
This is all great technology you are going to need to educate your users to use this. The big biggest single problem that I see with this is that this idea of sharing tables between multiple datasets is going to need advanced planning and our end users are not very good at advanced planning organizing amongst themselves, they will just carry.
All of the data multiple times, So what I suggest there's a little bit of Education. Maybe create some data flows for them look at the existing data for sets their creating and suggest where data flows could be useful show how data flows can help refresh times. An maybe restricts access to your data sources.
Create data flows Willy nilly or datasets, Willy Nilly, but force them to use data flows where they're available very last thing what's in the road map. This is all still preview.
There are going to be new things that will be added power query online will work better. It will have all of the functionality and data sources of power query on Prem. We've been promised some mysterious performance and scalability enhancements. I don't know what they will do we're going to have a better story around refresh because at the moment refreshing a data set is?
Disconnected from refreshing a dataflow, but really what we want is that if we refresh. A data set that uses a dataflow. We want to make sure the data flow has been refreshed before that happens that will happen. And we should be able to create and consume data flows through the API power BI API and looking at the car.
Data model specification, it has all kinds of interesting stuff like for example, you can have an entity which is partitioned.
And you will be able to have one entity that contains multiple CSV files and so I think there's going to be some very interesting stuff there for doing incremental. Refresh in our own way. But that's me that's my time up. Thank you for listening. Hopefully I've got you excited enough to go away and check out data flows and if you got any questions, I'll be hanging.
Here, or whatever and don't forget to do your feedback forms. Thank you very much.