My name's Paul Andrew. Thank you for coming. Hopefully we're all here to look at data factory butts going beyond the basics. We will now want to talk about complex orchestration. Deep dive into the service and look at some dynamic data factory pipelines. This is why we're all here right yes good.
Yes.
Before we do that.
You know what time it is we need to play the issue or icon game now for those of you that assume you speak before I've labored this and had a bit of fun with it, but I feel it's actually particularly applicable to this session 'cause. There's a lot of design patterns. A lot of architecture diagrams in here and I think it might.
So what I've done is I've taken my icon game and I've given it a where clause so we are going to look at icons, particularly applicable to this session. So hopefully this will kind of set the scene, a little bit so audience. Participation is mandatory. I'm afraid So what is the first one.
Data factory yes, good start.
Excellent good group.
SQL TV yes, thank you.
Data warehouse, yeah, awesome.
Virtual machines good one.
Not express route.
Sorry. There's multiple people I'll give you on Azure Active Directory. That's what you all said right.
Venetz Yes, a virtual network.
Was that one is not coat anger?
That is expressed route yes.
Already said it.
They like Gentoo could be yeah, as your storage account.
What about this one?
But shared a Muse play before.
What about this one automation? Yeah, good work and it's Valerie lightning bolts?
Functions excellence or a good group.
Logic apps, yeah.
Key vaults excellence, we're getting through is quite quickly.
Caffeine the one that underpins all with your service is excellent. Thank you applying the shorten version of the Azure icon game, you win nothing as usual. Now let's do some serious showing.
OK, back to the main slide back then. So yeah, all about Maine. Next, the slide deck. There is already a PDF of it's in my github repo. The community event repo that has some subfolders and things so feel free to help yourself to that. This is very much the one to take a picture of.
Picture of the rest 'cause there's a PDF of it in there. We all good. They don't want me in the picture now. I don't think so. I've been told I need to stay in these lights for the camera OK. We're going to talk about then well in this session here is hopefully how I've broken it down for you.
Very quick recap of data factory, then we're going to go with a much deeper looking at custom activities. Extensibility parallelism bootstrapping. The difference integration runtimes and how we can scale them out and then a bit of framework stuff at the end, so good.
Good OK, so data factory, then, who's used it.
Good most of you whose version one.
My sympathies go out to you people.
So I also use version one, I've been using the service since it was in preview in 2016 early 2016 and I think when I first started talking about it and I guess kind of the reason for this talk and why it's evolved is that the service as you know has evolved overtime. The version one in Denver.
Long and the Microsoft are continuing to develop the service for very good reason because in Azure. You know if we want that orchestrator of all things of all data flows is the service that we need so I've done. Lots of talks on data factory and I started off with one talk and that sort of one talk is now we've.
He talks just because we've needed more detail more features have been added. And lots of things have now included in here that we need to understand basically so an evolution of the service ultimately is led to an evolution of my talks and this is kind of the latest version, where we going to say much deeper now for those of you that don't know all those you.
Data factory is a very quick recap. It is the orchestrator of all things that we want to do it is the bossy service in Azure. It just tells things to go and execute. It invokes those services, does none of the work itself. It just has its him sheets of instructions.
Strikes the Azul Service Choir is it's a bit of a labored metaphor. But it will sit there as the umbrella if you like telling those other services to do things and as most of you probably know based on their response earlier. We have 5 main components within our date.
The link services that do the authentication where we connecting to our datasets that describe the thing that we want to interact with you know the file path filename the structure of that. That sort of thing. The activities that actually have the information tailored to the service that we're going to go and invoke.
Lines, which rap-up those activities into sort of logical groupings. If we want multiple activities within the pipeline and then of course, lastly are triggers the things where we actually want to the pipeline to fire whether it be scheduled or event driven or using tumbling windows or whatever it's a very quick recap those.
Control flow level within our data factory service now for 4 of those components. We obviously have our JSON and for those 4. We can make that JSON very dynamic excellent. This is kind of the starting points of where we're going to sort of go more.
Dynamic capability that means that this service is far more flexible than when it was first introduced back in 2016.
So those are the 5 components for greater dynamic and then very lastly kind of are often just to cover off the recap our integration runtimes. The first of which is the Azure Integration runtime. It can be used in a particular Azure region. If we want we can have the orchestra.
Sign one region and it will go and use the data movement components within another region that is your integration runtime will allow you to do that, if data factory isn't quite there. Yet when we get one of these out of the box by default within data factory 'cause. We have to have something that's going to tell those are the services what to do.
Our SSIS integration runtime, yeah, we're going to talk more about this bit later, but quick recap. It is tide to a particular region. It's that sort of cluster of computes that sat there to execute our SSIS packages and then the third version of course, our self hosted integration runtime.
Sing the acts very much like is a gateway if you like a VPN clients. That is going to allow data factory to reach down to maybe some on Prem services or go from a platform as a service type orchestrator to get into some infrastructure so let's dive strange with demo showing.
Let's set the scene were not messing about in this session, so for my first demo and I've got the architectures of this in the slide deck as well. I have on my laptop a self hosted integration runtime.
And what I have is a SQL server. DB Soria and Azure SQL. DB Let's use the correct name showing and within here. We have a file list table.
And we have an upload lock table now as we can see in the file. List table very simple structure. I've got a source file path target file path and some filenames or good.
So then in my data factory pipeline if I want to upload those datasets done something hopefully fairly simple given the this audience. We've got a look up activity. That's going to hits my file list table. I've got a for each loop, which will allow me to paralyzed.
And within the copy activity itself, so that's taking the array that look up activities providing to us is going to iterate over those values that I had in that SQL DB table and then my source in my sink within this copy then become dynamic using?
To it from that array.
And then lastly what I've got is a child pipeline call now. This child pipeline is just going to a bit of boilerplate code. If you like to send an email alert and my email alert is wrapped up in Azure function. That's just going to use the SMTP clients to hit in Office 365 subscription send me an email.
When I created this, it's using the web activity, which are now accepted your functions are first class system in data factory so we can have a function activity directly, but I say at the time. I did it with the web activity, so that's my first pipeline, which I say hopefully we're going deeper in the session so.
Quite complicated, but I want this to be the starting point for what we're going to do next.
So when I run this in debug mode. If I look at my output window. We know it's going to firstly hit my SQL DB.
Slowly.
And the Outputs from that is going to be that array of things from my table that we're going to iterate over and as we probably know already that for each loop, then will actually execute things in parallel so, although it's called a for each and we understand what that sort of recursive operation would normally.
Will actually do those things in parallel by default and by default. It will give you 20 parallel threads of execution so all those child components that are within the for each. It's going to run at once so that's what you can see here for those 7 datasets, it spawned 7 parallel copy activities.
Files that are on my laptop that are coming up through that local integration runtime.
Well, I also have within the for each loop you probably saw restore procedure that was being executed, which is just writing some logging table and then lastly if that complete successfully is just that child, pipeline, which is going to send an email.
So, in my outlook, we have pipeline succeeded who wears some important details in it, so.
That's the first demo.
So, in here, I say really fewer benefits takeaway. I've just got that architecture. So to recap and I say hopefully to make sense of where we are let's make it bigger.
So I've got that self host of integration runtime and that's running on my laptop data factory bit of metadata from my SQL DB copying it up to Azure data. Lake Jem, one in this case, and then a child pipeline, which spawns that function which hitting Office 365.
But those activities very much like that makes sense. Hopefully, that's a good starting point for this session here good that was the first demo next one.
What I've done is I've recreated SQL server replication with data factory? Why wouldn't you so in management studio, you probably saw I have an adventure works OLTP database.
And I've got another logical instance here as well with the same adventureworks. OLTP database same set of tables now in the target. One those tables. I've truncated the mall. So they're all empty so my customer address table has nothing in it.
So then what I've got is in my data factory pipeline. Same sort of idea. I'm doing a lookup activity, which is going to hit that first. Sorry it's going to hit that source SQL server instance and I'm sure this SQL query.
Is not?
The target table again just in case and a copy from one SQL instance to another.
Now what you might notice is that those datasets and things there all parameterized so let's just kick that off and what you see is from those pipeline parameters and what might become clear why this is running is that for all of that interaction from that so.
To the target and using that look up activity? What I've done is I've done it with a single link service any single data set so you know gone are the days of old where data factory version one where we had to have a data set defined for the source and the target. We had to have a link service defined for the source.
Yeah, it's very much a one to one mapping if he wants to do any of this stuff in V one. I was you know much further down the line. Now this entire thing can be dynamic so.
Say my dear, the same thing has been executed or within the shore this time, not touching my laptop so back in Management Studio. My customer address table is now populated nothing too exciting, but I say really the points. There is the fact that for this.
Demo architecture.
Which is definitely not complicated but I say the point really is the single link service single data set now that obviously proves that if we're going from 2 separate instances, and that for each loop is executing in parallel that single link service with its dynamic instance dynamic.
Dynamic credentials that runtime it's obviously doing the work behind the scenes to make sure those things are very separate otherwise. You know with those parallel threads that we've got in the for each we would get collisions. So that was just something that ideas when those links services first got announced as being able to be dynamic. I just needed to prove.
Yeah, that makes sense good so far, so that's kind of a starting point, then so that's kind of move on. We've sort of set the scene with what data factory is we've had a bit of a recap and we've looked at say some starting patterns that are quite dynamic, so now let's go a bit further. Let's look at extending.
Because we don't live in a perfect world we always have dates are that has problems in it, and corruptions and you know who has to who has had to deal with a CSV file that is text qualified that has a free text field in it, and then some nice.
Carriage returns in that free tax field, which is broken. The structure of the CSV file. Yeah, we've all probably had that pain now.
By default in and with the sort of link services and the definitions of the datasets that we create. We can't deal with that. We have to do something more. There's always these edge cases and particularly if we think about all the wonderful stuff we could do within SSIS. We have script components and third party people would cry.
Say, the 80% is not the problem. It's always the 20% that's the killer. So the way we handle that firstly in data factory our option here is a custom activity. Now, before we go into water custom activity is I just feel like I need to tell you what?
Services because the custom activity will run on the Azure back service underneath and the back service is just a pool of computes very much light. You would define a set of virtual machines in Azure. The batch service wraps up those virtual machines in this compute.
And then one virtual machine within that pool becomes OneNote and this is quite nice because it's wrapping up infrastructure as a platform as a service offering which and it means that in our batch service that computes pool can autoscale out.
More work to do it allowed more compute nodes. It will do that is quite nice. We can also. If we want RDP to these compute nodes so if we want to actually debug it. We want to get into it and see what's happening. We can do that as well so that's just the Azure Batch service bit of a digression, but I think we need to do in the standard it is when we talk.
Services.
So then the customer activities themselves well let's look at how we create. One then how do we build a customer activity. If we've got this? Horrible sort of CSV file that I described with that carriage return in well. We need visual studio that good old thing that.
Been using for awhile that isn't browser base that does work on an airplane and we make some we build some custom classes in here in C sharp or whatever. We want really, and we basically need to compile that custom code down to an executable and then what we do is in Azure.
Start executable in our blob storage.
And then data factor comes along so we've done that development work and data factory needs its back service with its pool of Computes Yeah.
And then we have our storage so we have our actual data. This nasty CSV file. That's got those carriage returns in and within our data factory. We build a pipeline and that pipeline has our custom activity in it, it's got all this information and then.
Happens is it runtime data factory will go to the batch service and says. Hey, I've got some work for you to do it will say going get this executable from blob storage and put it on all of your compute nodes you will do that for you and then.
Service runs the job so it will get that raw data. It will download it onto the compute nodes. It will do that transformation or execute that custom code as we need it, too, and then after it's finished it will put the clean data back.
So that's a way that we might need to deal with that particular scenario that I described and it gets worse because this back service 'cause. It lives there in Azure is very much a separate thing today to factory. It has to authenticate against the storage itself as of September last year it.
Inherits the credentials from our data factory. It has to authenticate by its own means so we need to use key votes. We need to have some credentials in key vault for our data. Lake store that the batch service can go away, and use and then after it's done all that.
I think it will do for us quite nicely is our standard out in our standard error. It will write them back to the blob storage and the Guids. That's used in our data factory pipeline for the execution. It will create a folder within our blob storage and that's where the debugging info.
Now you may think Paul YV drawn this pretty picture. 'cause this seems like hard work right yeah. So who has created an SSIS package and who has put a script task in the SSIS package and we're in some C Sharp.
I would argue this is the same thing except when we did it in SSIS. It was all wrapped up and it was all nice and cozy and fluffy in SQL server, and we had that SSIS package. We put the script component on and we did edits and it will give us that sort of temporary visual studio environments where.
C sharp we were close it and it would go away now, what happened is that SQL server would take that C Sharp. It will compile it. It would put the binary in the DTS X file if you looked at it in notepad and then obviously it would get ran by SQL Server in our SQL agent, so all that Fluffiness and all.
And that we were comfortable with in SSI ass if we want to achieve the same thing in data factory. This is just the the method that we have to go through to do it and because we're now working in Azure. It means that the plumbing between those platform as a service that is platform service tech we have to do that work.
As well.
But it's possible I've got a blog post on doing this and I guess the very last thing I would say is that those compute pools now can access Thavenet And if you've got express routes. It may be a consideration, so that's a going down a bit of a path with.
Custom activities and extending data factory by that means if we need to.
Next then what else can we do to extend their data factory well. We have the ability to use to hit rest. APIs everything's gotta rest. API now right except my wife no rest API anyway.
We have within data factory the ability to use web activities and with that web activity. We can define what we want to do so again to post to put and we can authenticate we can use. Barra token authentication in here, but you kind of have to dynamically inject it into the.
Call them relying on the actual authentication settings, it's possible.
So as another means of extending our data factory if we have 2 and you may have noticed. I've crossed out functions because I said earlier that functions now have their own activity. There sort of first class citizens. Now we don't need to hit them with web activities, which is nice now. The reason for mentioning this and thinking about it well.
Custom C Sharp code that we need to execute then we could say Well, let's just do that in functions. It's far easier and all that stuff, you just talked about in the previous slide well. The problem in the argument. There is that those functions are only probably going to execute for 10 minutes and then you know they will die, so.
Of data that you need to crunch through you need a sort of longer running process that you have to manage. This is where inactivity custom activities required over maybe something like hitting Azure function, with the same code. In it, food for thought. Lastly, then if we want to extend our data factory we have.
We want to run those packages. We have this you know it's not like our backs service, where we can like get to it, and we can RDP to those nodes. The SISIR is kind of wrapped up in this veneer where we don't really have the same visibility, which is a problem.
And just thinking about that problem actually when it comes to paralyzing things and actually we're going to use our SSIS packages because.
If somebody is put a lot of investment into those sis packages on Prem then absolutely let's lift and shift and get them into issue. But it's not going to be for free. And it's not necessarily going to be easy. I would argue that we could re engineer. Those packages to use cloud. Native Tech, which might scale better, which might be more cost effective.
Slight digression let's think about that. SISIR within the context of our hosted integration runtime because I demo'd the host of the integration runtime on my laptop getting files up there? Which worked but could we also do the same thing with Rs.
Well, yeah, sure you know if we install the Azure feature pack. We could just do a very simple copy from A to B and actually it might work better, so there is a design decision here because our hosted integration runtime can have 4 nodes.
Where is RSISIR can have 10 and we can actually have 8 packages executed in parallel per node so potentially you know if we're getting a lot of data from an on Prem source, I might actually say. Let's create an SSIS package to do it. Let's have one package.
But yeah, we can obviously use bill or something to create them. Let's get them up there and actually it might scale out better than what we can achieve with the host integration runtime so bit of a design decision and like with our custom activities that pull of VMS will and can have access.
And it can use express' roots so you know bandwidth considerations as well. So sticking with that and let's pretend that OK. Let's have 10 compute nodes and let's say we have a package is running per node. So we've got these 80 packages that we need to execute now.
It can be expensive right somebody's going to pay for that so.
And part of the reason for this talk is that we're not just designing for the right solution in terms of data engineering anymore. We've got a design for cost because Azure is expensive so if we're going to have 10 compute nodes. We don't want him on all the time we do need to turn them off at some point and.
So how can we go about doing that so here comes another one of my long winded pictures with lots of arrows so let's get into showing well. We can turn off that host integration runtime with a bit of power shell? Do we just employ somebody to run that?
We could wrap that power shell in Azure automation, yeah, sure and we can give that run book a webhook.
It's good that it rhymes and then actually within our data factory pipeline. We can hit that Webhook 'cause. We've talked about it already that within that extensibility so we have a web activity up there that hits that webhook URL which starts our integrate.
Brings up those eights those 10 compute nodes and then we run our SSIS package is that good enough.
Now of course, is not good enough were better than that.
What else should we do then now this thing takes 20 minutes to start up. I know right about 20 minutes so maybe we need to include await activity up here and what do we do? Do we say wait about 20 minutes? Do we have this notion of about. I wish we did.
Fair enough or we better than that, yeah cause. We are so maybe we change that weight activity. Then we using until activity wait until the things actually started seems reasonable logic right, but how do we actually figure out when it started?
Because the web activity that we told to start is going to be an async call it's going to fire him forget. It's not going to wait for the room book to come back and say it's finished just be going run. It so we need to do something else, so within that until activity, we now are going to have to have an uh.
Activity and we're now going to have to hit the Azure Management API.
That is your management API is going to be needed to figure out if the particular execution of that room book has completed or not the power shell that's in it. Mercifully and the command. Let's that we use for the IR is synchronous so that's OK, so that room book will fire.
Will say running and then at some point it's going to come back with completed but what we need to do is if we're going to manage. This properly within data factory. There's a consideration, there've now needing to hit the Azure management. API just to figure out if the thing we're trying to turn on has actually started or not before we run out.
Just so we can go back to the customer and say, Hey, your bills not as big this month look what I did now that's going to need a bear a token because she can't just hit the Azure management. API Willy Nilly, probably something to do with security of who knows? But we have to go and get that bear a token.
Sort of short lived token it doesn't last long and that thing is then going to go away and come back and we need a service principle. Of course to get that so we can pass that into our until activity use it within that inner web activity and then we can figure out if the things started.
Yeah, people a little bit tentative last thing we could do because I've got that wait 20 minutes there. Do we want to sort of iterate over every single time and say if it hasn't started wait 20 minutes? What we could do we could also include in if activity so if the thing hasn't?
Wait, 20 minutes if the thing has started return straight away just means that you know, we're saving that last check on the until recursiveness makes sense good and then last but not least. Maybe it might be a good idea to stop it.
Think so, so.
We've sort of scaled out which is really quite nice and we get those 80 parallel copies. But there's say the point of this session was there's more than meets the eye when it comes to orchestrating these things because we do need to design for cost so that's a problem now that pattern work.
But there's a fundamental flaw here.
Because my Rune SSIS package activity that's just one package or pulled silly. We've got 80 packages that we want to run that's only going to do one we've got another problem to deal with him and we've got to scale this thing now well how we're going to do that.
Now, as you already saw in my demos. Let's have a look up activity. That's going to hit RSISDB and it's going to return. Those list of packages that we've got deployed and then we can use a for each loop. So we try over them coming and it will scale out and it will run them in parallel? Yeah.
Now the problem with that is that the for each activity is only going to give us a Max degree of parallelism of 50 threads default is 2050. If you force it. So we've still got a sort of disconnect here between 30 packages that aren't going to be run when we particularly want them, too.
I'm going to cloud solution if it's not going to scale out as far as we wanted to So what can we do about that? Well, this is where things get a bit more fun. So let's Park that components that understanding of a look up for each and then call the packages that's going to allow us to.
Packages we need the extra 30, So what I'm going to introduce is a second SQL. DB just for the simple fact that the SSI SDB is obviously kind of a closed box. We don't want to play about with that, too much 'cause it might get changed so will have a second metadata.
And what I'm going to do is I'm going to build a pipeline over this thing and what I'm going to do is in this pipeline is firstly. A copy activity, so like I showed you my second demo. I'm just going to go and get all of the stuff out the SSI SDB into an environment that I can control. I want the information.
Which is a diploid the folder the project that they're in and I also want the information about when they execute and how long they took if that information exists. You know, we can just do a left join into the catalog views to get that and I'll have then within my metadata DB that I can.
Stored procedure that is going to have this notion that I'm going to call book. It's maybe a better term for it, but will stick with buckets. For now, so I'm going to have a stored procedure to assign packages to book. It's and what I want to do is I want to have 50 buckets and I want to say that.
As I've got deployed look at how long they took to execute order them and then let's round. Robin those packages in those 50 buckets, yeah, or you know by some other method. If you want just so we have, then this notion of book. It's that sits above and those buckets can then execute so.
Pipeline and then get another for each activity, so this for each activity is going to scale out, but not iterating over packages its rating over book. It's so the first level is hitting the book. It's and then within that parent for each and then going to have another look up that.
It gets the package ID or something some sort of internal identifier and then a child pipeline. So then that child, pipeline that child execute activity pipeline is then going to call what we had before makes sense little bit.
OK, good so we had a Max topper 50, here we've then got a Max top of 50, up here. But obviously you know with those book. It some with that additional level of Scaleability. We've now achieved a better Mac stop now 2500, I'll do right good enough.
That's now going to mean that our 80 packages. We can actually execute them in parallel. Yeah, good now. The other thing to consider is that because we're using the execute pipeline activity. That's something that can be synchronous or asynchronous.
So sort of bank through them and just blast them all off, you can do that. That'll work but just bear in mind that there are actually some limitations of the data factory service, which I'm proud to say, I've hit most of them.
I think last achievement, you know, particularly the one that's on here about concurrent pipeline runs is a problem. This isn't actually in a Microsoft document. You've got a sort of digging through this. Github repo to find this table. I don't think it's widely publicized.
Fun things about 50 data factories per subscription. No, I've not heard that one actually but you know just bear in mind with this sort of setup if we're sort of building for this sort of parallelism, then you know it's a consideration now.
Is that it did we stop there?
Do we want to go again? Yeah, course we do so. Let's Park that and then the last level of pipelines. I want to talk about going to call these the grandparent pipelines because in the previous slide. We talked about starting and stopping our SSIS integration runtime, yeah, we just need to.
So while says that with our grandparent level pipelines. We can now start the integration runtime is kind of like one bucket of stuff. I've never use the word book it. We can ruin our packages, which is obviously going to do this thing that we now understood.
All that good stuff about power shell into the management API, you know, we can wrap that up somewhere else. So I guess what? I want to say is that in summary in this pattern applies not just to SSIS. I kinda just used SSIS because of the integration runtime, but at the grandparent level. We want to really have something that.
Control it or kick it off, you know, maybe some wall clock trigger or something like that. We might put it the grandparent level. Then in the parent level. We want that component level control, stopping and starting those other Azure services, or whatever. We may need and then defining those book.
Can sort of achieve that scalability that we need as I say doesn't have to be SSIS packages? Do we store procedures? Do with you SQL scripts. Once Upon a time and then finally at the child level is where we're actually hitting those services were actually doing the work makes sense good.
So decently similar.
Is mine better?
Totally different OK good so hopefully that makes sense and kind of the crux of why wants to talk about then. Let's move on. And just think about. Some other sort of production considerations in data factory when's the session finish?
OK, 10 minutes good stuff, so bootstrapping data factory.
These data factory going to be the thing that we used to bootstrap or other stuff within Azure doesn't have to be we've got some other services. Here, as well, which I've already mentioned now data factory will call functions run books in logic apps, but also functions with a bitof.net can call data.
Room books with a bit of power shell can call data factory and logic apps. His actions in it as well to call data factory so you know, we don't have to use data factory to bootstrap everything I mean, I would 'cause I like it. I'm biased. But if you've got some other department's with the developer considerations to consider maybe.
Use logic apps to do some bootstrapping and then start date factory.
Just some options and you know if I sort of extend that of well. Why I had a customer that actually had data factories in separate your tenants. You know they had sort of very disconnected process is we had to kind of bridge that Gap and make sure that we had some sort of master thing that was.
Maybe we have data factories in separate subscriptions.
Maybe even separate your regions you know if they're in the same subscription is not really a problem. But you can understand. Maybe if we're going to really sort of push the envelope here, we might want to sort of build more of a global architecture that has these factors and then maybe we need functions in there as well.
Who knows just sort of extending that and thinking said bit more about bootstrapping I had a customer that we built an analytics platform for them is a very sort of simplistic analytics platform, because they were really keen to move to the cloud. They said that actually I don't just want to you.
Sorry for my analytics platform, I'm actually going to look at the data ingestion peace and what they did was they had like a SQL server, and some other orchestration. Cron jobs and all sorts of things and they said. Actually, I'm going to use data factory on Prem to replace those schedule.
Down here as well, and I'm actually going to have data factory to manage everything. So yeah, great. I'll build that for you so we did you know we replace those on Prem things with data factory because we could now another customer instead of data factory they use logic apps sure logic.
Has its own gateway that will reach down? It will detect on premises file events. If you create a file in your's and then data factory will maybe use logic apps to trigger it and go away, and through that just another consideration, touching on our host integration runtime as I've said.
Have the option to have multiple nodes, so we do get that failover and load balancing but what if we use an express route. If we use express route. We need to change the boundary and what seems a little counter. Intuitive is that we actually need to have that integration runtime on a virtual machine in Azure and then that.
Access to Vineet Access Express route and get down, there, so say just another sort of pattern that's come up with my experience. The service so far that you need to think about.
So bit of a sub agenda, lastly, I just want to touch on.
Processing things in more of a framework type way because you know, we all love a framework that we now we could build a data factory and we could have sort of one to one executions of pipelines and things and that's all good. But we can do better. We know we can do better.
Relationship between our parents and child pipelines and let's have AQ that's going to sit in the metadata database and we can pull that Q Every 5 minutes and then we can. Maybe have a functions layer that goes and gets information about that metadata, maybe let's have some blob.
Such East or the schema definitions of the datasets in the data frames that we're actually going to process. Let's separate the python scripts and then maybe you know. Let's have a grandparent pipeline rights to that processing queue. We can do these things this is good.
I've also talked about data bricks there, but actually databricks has for me. Certainly had a quite high barrier for entry because I didn't know spark but I didn't really know scholar so you know that was a problem. I was much more comfortable with the relational stuff, but then if we transforming data. Those are kind of got their own problems as.
Maybe we using hates tunes site but it gets expensive use equal future uncertain custom activities. We talked about a necessary.
It has no future, according to don't quite made quite this lady over here, so I'm sure if you attended the session before mine. We talked about dataflow and as we know that dataflow now within data factory will right that's color for you when it will pass it to Databricks and that's all good. I'm not going to tell you what?
I'm putting it talks about it, so far and that will access are expressed through the point for me is that where does this feature in our wider design patterns because.
If we've got our data factory now we've got our source systems and we're bringing that stuff up. There then is dataflow going to feature is one of our sort of child or parent pipelines. Yeah, we're trying to create our data here. I mean, they say this is food for thought so I'm not really sure what the answers yet, but this is kind of what I'm looking at the.
And actually do these 2 switch places do we use dataflow for our sort of simple data warehouse loading because it's got that? Look up activity so that look up transform within their you know, and maybe we use data bricks for the heavy lifting stuff first.
You know, maybe well, what about you know if we want to keep stuff in memory. You know because why have a spark cluster. If we're not going to take advantage of those in memory capabilities again food for thought. Maybe you know say we're replacing that or you know that data flow that we've now got we know.
Datasets dynamic with paths and things is there going to be a situation where the Jason that actually goes into those data flows can, we make that script dynamic as well. Can we start metadata driving the transformations at the data flow level as well as the control flow level food for thought.
Yeah, watch this space so that is, it for me folks as I've said the slides are in my github repo. Thank you very much for listening.