My name is Xiao Fang. I am at the senior program manager from Microsoft in sitting in Redmond. So my focus today or actually my job. It's to manage the open source databases in Azure. So we have actually 3 offering right now 3, database engine. My SQL, Postgres and also Maria DB so that's actually.
If you have any questions about this presentation or any of your use case scenario for your company that you can actually talk to me later or actually email me. Let's actually get into can you guys hear me properly I can not really OK? I think that's good now so a little bit of the.
So let's talk a little bit about the open source and his opportunity right so open source has been around for a long time and compared to all the license product. For example, SQL server. Oracle many of the modern developers, they actually develop the?
Workloads on open source will talk a little bit about what is really open source is Andy opportunities right then will explain on the the Azure managed databases which are all the open source database engines and what are the use cases that are really?
Or some of the really advantage of using open source is and the last topic. I want to cover is how do you actually bring in data into Azure if you're interested to actually use Azure open source databases?
Now let's actually talk about open open source and the opportunity. So this is actually from the corner and noise research right here, so more than 40% of the CIO will have some kind of open source strategy in D in the planning right now. This is actually like I say on top of the.
Yeah, they run their enterprise workload more and more of the CIO actually look at open source because of the vibrant community because of the developer base and also the without vendor, locking right? That's actually the most important point now without paying for the license and without the vendor locking how can I act?
Source to enrich my and also make it easier for me to manage my entire portfolio of the software in my company now more than 60% of the developers actually wanted the cloud providers to involve or very active in open source community like I said before 90% of.
Surprise developers they are actually using open source today. That's again like, I say the key point is that without paying a hefty prices for all the license product and also with the maturity of the open source databases. They want to actually use open source.
Now, what is Microsoft journey into uh open source so far, so Microsoft has been involved in since April 20, 8:13 right when we offer. The Linux VM right so Linux Linux. VM is actually the equivalent of like the windows VM that we actually offer in Azure right now.
Not going to go through each of this, but Microsoft continue to invest in open source until today right now. If you guys actually follow Microsoft commitment into my into open source. The last one side's acquisition anyone of here actually notice the uh.
Date in February or actually January on the cytus acquisition, so sithis is D Postgres Extension company than Microsoft. Recently, acquired so it will actually enrich Microsoft Azure to be able to short your data from a single instance in multiple notes and extend your workload.
Single instance of a restriction to multiple storage and also the performance so that scientists acquisition is our latest engagement in open source.
Right so more research data actually shows that the adoption of the open source. It's actually it's growing right by 202270 percent of the in-house application will develop actually owned open source is this is actually from the open source Market Garden.
No, the table on the right is actually showing you in terms of the popularity that we actually gather the data from Stackoverflow No. I say before open source is actually building community and vibrant and more active of the community that means the more mature.
Right now, Postgres is the number one relational database in terms of the popularity of the databases in open source right now in this choice right here. It combines both the no SQL and also relational but Postgres continue to be number one for the recent few.
On open source database engine. Now then let's actually talk about the Microsoft offering on the manage open source engine. I have a little bit of test or quiz and see whether you guys actually listen to what I say before what are the 3 relational databases open source engines that we offer?
Mongo Mongo is virtually no SQL offering right, I'm relational databases.
Maria DB yes, anyone Postgres and what is the last one.
My SQL there's actually something that away on one shirt. Right yes, so Postgres my SQL, Anna Maria. DB is the the 3 engines that we offer now. I'm going to go a little bit into more of all those 3 engines, so like I say we offer 3 engines right now, we have.
Very rapid growing off our database or our customer base. We have 29,000 and actually growing right. Many of our customers like Adobe. BMW has been our customers since they want right now, we are actually diploid the the the we are actually offering those days.
38 regions in the entire Azure right. The Azure Data Data Center. Now we offer them to more than 100 countries right so some of the for example, government cloud in US or China. We actually have those offering as well.
Market.
I think one of the data data center that we have in UK is actually UK self, which is actually an also Western Europe. Those are actually 2 of the biggest regions that we have in terms of open source now like I say this are actually all the customers that we have and we actually rapidly have more customers. Joining us just because.
Turn into open sources.
Now, if you
If you leave today there's actually one single slide. I want you guys to actually take home right and I know about this. This are actually all the value prop that Azure. Uh open source is database actually offer. I'm going to go a little bit detail into each of those silo, but we are offering the community version of.
A database is now you're going to ask me. Hey, what does it mean by community version and why is it a one of the advantage now community version is the community version that is actually free for download that is not someone actually for can make it like a for example, enterprise DB is a fork.
Postgres they will add a little bit on top of it. They want to sell you on the support and also the license of the feature but we are offering the community version. We actually put it on Azure. And then we actually manage all your backup and restore so that you don't have to spend all those time, managing your databases and.
Port and key point is that there's no vendor login so lock in right so that means your community version of the database engine will belong to you and then you can actually move it. Let's say for example, you want to migrate out from Azure. You can actually move it to somewhere else as well.
No uh built we have the billing AK which I'm going to go talk a little bit detail about an we offer for 9 ceceli right on top of the paycheck. It'ST most secure and also compliant platform. Azure is the most secure and compliant per platform. We continue to add-on certification.
Our in our repository for the managed databases. We have building intelligence. We actually share the same platform as SQL, Azure so. That means some of the IP for example, threat detection. We actually offer that as well in a Postgres and also my SQL, we have.
Most integrated Azure offering, we understand that, your application is not database alone. You have the front end. You have some of the radius cash, which actually we tightly integrate into Azure right in our offering.
Now let's actually talk a little bit more about the details like I say the manage community version right so we offered a free those are the version that we list right here right there. We also offer the automatic updates right. The minor version. Update we managed for you know, I'm sure you guys actually manage all this upgrade like you know the patches.
The major version upgrade you guys have to test and actually put it down and I also like you know diploid them. We actually manage all there for you. The minor the minor updates, Azure as the platform actually manage your database version for you right. We also have managed the security fixes so that you don't have.
To manage in all the operations.
Uh.
No, the built in a cake. This is actually one of the major advantage that we offer as compared to other cloud providers. For example, AWS and also GCP right so we have building availability of 4 ninths. SLE now, what does it mean so you can actually diploid your database?
So data center right and let's say for example, if we will have. We will deploy your data into 3 replicas. Let's say for example, one of the data get corrupted.
Your replica we will actually automatically create another replica in the same data center, so that means your data will always be up and running right now. Let's say for example, your DB engine actually go poof and get disconnected right this actually what happened.
So what will happen because of our design. We actually built a database on the container. That means it will actually get spin up really, really quickly in the matter of 30 to 45 seconds right now, we have a management ring that is actually in our architecture design once we actually detect.
You search leaping and also test so and we actually detect your database engine is disconnected. We quickly spin up another engine right in the middle of 30 to 45 seconds and get connected your gateway will automatically connected to the new instance right here right now, what your app.
To do is keep on retrying so that's why we say we have the built in high availability for you and then we guarantee 49 SLE.
No, that's actually address the white elephant in the room. How is this compared to a WS right now. This is the major difference if you actually look at the left hand side of the chart in a single data center. Amazon RDS actually do not guarantee any SLE. In fact, your database go down it goes down.
It depends on like you know, God know how how much time is going to bring up right and then you probably have to contact the support and actually get the database up right now in Azure. It's a It's a different design that with the replica. We and also the containerize database engine. We can actually spin up the engine really rather quick.
So actually get your application up and running now. Amazon will offer 39 or actually 99.95. SOE if you deploy to multi AZ. That means if you have a yeast us.
Or then you have a single AZ, then if you actually diploid. The East US 2, which is multi AZ. Then they actually guarantee a 99.95 SLE now. This is actually not true for Azure right. We even though you deploy to a single data center. We already guarantee your phone SLE.
No so in terms of compute I actually stop price right here. This is actually in terms of compute of AV CPU and or V core in Azure. We are actually twice cheaper just because of a WS RDS actually require you to deploy into 2 sites, which is multi AZ to guaranteed.
SLA but Azure don't right now, so that's why we actually twice cheaper. Now let's actually talk about storage now storage is the same thing we they don't in terms of storage. They will a WS actually in terms of the general purpose storage the pricing is up there?
In the single AZ. We are offering the comparible price right now in Multi, AZ same thing you're actually incurring double of the price in a second data center. Azure business model is very simple. We actually just charge you a single data center.
Storage now, what about IOPS now you guys will say that, Hey, you know, my application or my workload require really high. IOPS AWS actually offered something that's called provision. I opt you can actually get a 4 terabyte up to 10 K of IOPS. If you pay pay that means you pay.
Higher just simply scale in terms of the storage will give you higher IOPS. Sorry we are offering up to 12,000 of I opt this coming April in selected data center. I think West Europe is already having 12,000 iota right now in private preview.
So again in terms of storage Azure is actually 3 to 6 times cheaper compared to AWS.
Like OK, so this, I already talked about, we have a lot of certifications, certifications, so regardless of the industry for example, if you're in healthcare industry. You need like HIPAA and we already HIPAA compliant right in terms of our database offering and we continue to actually get more certification.
Or services, we have global reach. We are 38 out of 54 regions that we deploy well. Why is that include some of the government cloud and also in in China right?
I'm not going to repeat the same thing, so this is actually a our databases. Actually, tightly integrated with all the Azure ecosystem, so for example, if you want to run a AI workload. We are tightly integrated with the AI services and also power BI right you actually gave you the dash.
What are the most common use cases that people use open source databases now in terms of content management services?
For example, you want to deploy AE Commerce website or you have a block right that actually go out that actually a lot of people read and write an comment and stuff like that my SQL is the most popular just because of the platform that is using right so for example, you will use what press to dip.
It's the most popular platform that you use to actually create a block website and by default my SQL is the back end, so some of the digital marketing websites because of the heavy content management my SQL is actually the popular choice so the next few slides is.
What are the characteristic and also properties if you actually want to deploy a digital marketing websites for example, you need like very fast and also fluid experience. You need to be able to scale elastically when the users are actually getting haviar or for your E comma so you have more work.
Christmas time for example, right so Azure offer. All of that elastically so that means you don't have to let's say for example, if you deploy your application into VM then you have to pre plan ahead of time and say OK. Now let's say for example, I think there will be like twice of the transactions coming in the?
We have to pre provision and fill out all the VMS ahead of time. But in Azure. You just scale it up. There's a bar for you. To actually scale up the the compute and also storage so that makes your life actually easier. So the next few slides will actually talk about this are some of the sample architecture like if you.
Ploy my sequel content management website. I'm not going to go into detail into each of this, you have a slight of this one. So these are some of the architecture that we put together or customer users if they want to deploy AE Commerce site.
So no apps right and also this is actually a fault tolerant architecture.
No so the next next topic that I want to talk about is OK. So you're saying that I'm very interested to do a proof of concept, and I already have existing databases or application. I have a little bit smaller subsets of data that I want to migrate or I want to test that out in Azure? How do I?
Data into Azure to actually get the proof of concept started.
No so we actually because of the migration is actually a very important topic. So we offer database migration service as the first party solution and database migration service is one of the migration service that we created to cater to many sources and many targets.
They are not only like managed to migrate your my SQL, Postgres as you can see over here. If you have on premise SQL server that you want to move to a VM or SQL, Azure or if you have on premise SQL server that you want to move to manage instance.
This database management or a DM S right here. It's the one solution for you to do migration right now. The gentleman back there, actually mentioned about Mongo. DB we actually do it. Mongo DB migration as well to cause Modi be right. You know wrote. Cassandra is actually the next thing that we will do migration.
No so then you're going to ask me. Hey, you know for open source is there are tons of tools. I can do a dominant wrist or I can do import export? Why do I actually have to learn another service to do migration? No there is a unique advantage.
We're using DNS right here. If you see the near zero downtime? What does it mean so that means your source? When we're doing a migration your source database continue to be up and running. You don't need to put it down into read only mode or actually a maintenance page come out that actually encouraged.
Your application right now, depending on your size of the databases term, sometimes migration. A dominant wrist or or a import export will take hours if you have multiple terabyte of your data. It will take days right so those are all down time now. If you're using the Ms because we call it online.
That means your source database will remain online now. How do we do that? I'm going to go into a little bit of detail right here right? I'm using a sample application right here. That is actually a shopping cart migration right. I have hosted the data into the you can put it in my soup.
Now then I establish a DM S Migration Pipeline Right. So when DMS is actually running what it does the first step is it will migrate the initial load?
So let's say I start the pipeline right now, so this is actually T one right. It will first move the initial load into about copy using bulk copy protocol right now, then anything that actually comes in so that means the new update insert or delete that comes in after T one, it will get replicated.
Azure right now, so how does the Ms actually know where is the new updates coming in for my SQL there's actually something called been lock so it would just replace the thing that is actually the log that is actually updated into been lock and write to Azure. Same thing for Postgres. There's something called right.
Wall lock right so we're actually looking into the log and replay the log into Azure until the day to catch up in Azure.
Now so when when you're happy with the migration when the data is all caught up then at that time what you do is you put on a splash page or a maintenance page on your application. Then you're incur a very brief minimum down time so you wait for the last batch of the data to catch up in Azure so DM S will.
For new updates an update and you that the latest last bunch of the data in Azure. And then you complete the database migration in DM S then at that point, the last operational step that you need to do is update. The connection string from on premise and point to Azure now. That's the time that you're done right, you.
So this is actually a huge difference. Now let's say for example, you're using a backup and restore strategy your down time happen in step number one right now, it depending on your data size who could take hours or days, but using DM S you only downtime actually start in number 3 and this is actually a very, very brief minimum downtime.
Right so you just need to wait for the last batch of data to catch up and you update. The connection string and then you're done so, sometimes we have customers telling us that down. Time is actually in terms of minutes. So it depends on your on your main maintenance or downtime SLE as well right on your application.
Right let me actually go into a demo of migrating application that I have using DM S.
Now, how many how many of you guys are chili familiar with database migration service or have you guys heard about it?
No yeah, so tomorrow, Rogers my colleague right here. We have in depth session to talk about database migration service, so today. I'm just going to show you a demo. I'm going to migrate application that is hosted on my sequel to Azure from from a VM to Azure.
Alright let me actually switch to my.
Azure Portal right here, so for those of you. I don't think you can see my screen. Once I mean moved it over.
Right.
Now this is actually not going to help me because I cannot see that.
I need to think this thing let me.
Alright much better.
Now I'm in a Azure portal. Right now right so for those of you who haven't seen or not familiar with database migration service, you can just go to create a resource and then you type database migration service.
It will come up as this one, so let's actually go ahead and create that one. It has the same diagram that I show you guys before. With the multiple sources and multiple target do read the documentation right because the documentation were actually explain what is offline migration and also online migration? What are some of the pre Ray?
To do in order to actually set up the migration.
Now, once you create this, I'm going to show you a few more workflow right here. You're going to give it a service name.
Which other can actually just say SQL bit?
Right and then you're using your subscription right. Here subscription is the entity of your Microsoft in terms of Azure subscriptions. It will try to your account. I can actually give it a resource group, which it can actually default or something that I have right here.
Location that means you're going to deploy you're going to select to a data center that you want right so let me actually do that again.
It shouldn't.
OK, much better, so.
For UK folks right here, we have, I think I have UK self and also West Europe that are close to here right so one of the let's say for example, I choose UK self and or UK. West one consideration is that you should actually put your migration service close to your source.
And also your target you don't want to move for example, your on premise database is actually sitting in UK sells you don't want to stand up a migration service that is actually somewhere in let's say West Europe that sitting in Amsterdam. Right now, you're going to incur a lot of data latency that you don't want to actually get impact.
So our recommendation is Deployer Migration Service in the closest region that you have with your on premise and also on the cloud right on the target.
Now this is actually a virtual network now virtual network is something that I don't have a virtual network right here, but let me select something that is actually in EOS.
Now virtual network is something that is actually Azure vnet, that is set up to have connectivity to your on premise. It needs to talk to your SQL server port or connect to a my SQL port right. I don't need you to actually open up the entire data center to Azure. But I need the database port to be uh.
To actually establish the connectivity to talk to your source now remember migration service is to migrate something from your on premise or your VM to the target in needs connectivity to be able to talk to your source, which is actually your VM or your on premise to the cloud.
Now so you actually will select this one that I already have DV net set up.
Uhm I will have the pricing tier right here. I'm going to show you what is the difference so I have a standard and also premium so if you're doing online migration? What we mean by online migration is exactly what I explained before your database remain to be active and then your data were just replicate over.
On that feature select premium so premium is the only one that allows you to do online migration right now. The standard one of the standard will offer you, you can actually choose one or 2 or 4 cores. It will allows you to actually do offline migration for SQL server. I'm going to choose premium.
OK, OK now then you will create the next step is actually create this service, which takes about 10 minutes to deploy right now. I'm not going to do that deployment because I already have one up and running now in this project right here that I have my name, it's equal bits.
Can actually do multiple migrations for example, I can do a my SQL migration? I can do already as Postgres. SQL so that means I have. I want to migrate from AWS or ideas to Azure. I can actually set that up, I can also do a sequel on premise to Azure sequel.
Value prop of DM S is actually a one stop shop to allows you to do multiple or different sources of migration.
Now so let me actually do my SQL migration right now.
My source is actually my SQL.
Right and my target is Azure Database for my SQL.
And I would choose online migration.
Now the next step is I need. I need to put in my source database so like I say before it needs to be able to actually talk to your source and migrate your data over So what I have right here. Let me actually grab the IP.
Of my VM.
So I have a application that is hosted here 36330, 6, is the my SQL port.
I will put in a password right here for it to connect.
Perfect so far, so good now the next one. I need to put in my target so this is actually the my SQL instance that you already deployed to Azure right, and this is my connection string right here and put in a user.
All right great, it's already connected no DMSO actually automatically detect what are the databases that you have in your source database? Now I can actually select all of this and move all of them together for today for the demo. I'm going to move only one database, which is actually the invent ory.
I'm going to save this.
These are some of the advanced setting that we have in the if you want to do if you want to speed this up. You can actually put more database or more tables into low in parallel right. But I'm not going to touch. This this is I can use all the default settings right here, then I'm going to say.
Right so this is give it a migration activity, and it's actually doing it. It's what is establishing right now, it's a migration pipeline already talked to the source in knows what kind of database? Is it has. It's actually setting up the initializing the pipeline right now right.
Yes, so looks like it's already initializing the database, the the migration pipeline. I can actually click into the database to actually get more details.
Alright perfect so far, so good so it's a It's a pretty small database with 2 tables right here. Now we already migrated the first initial of the full load already migrated 9 Rose and also about 200 rose right here. Right now, you can say that OK now. I'm actually pretty much done right, I already migrate the data or?
What I explain just now is where is the minimum down time because the new transactions keep coming in right? I'm not putting my database down so you're going to say that, Hey Show, Where is my my knew what if my new transactions come in and what happened to them now? I'm going to simulate inserts into the tables so let me.
Another machine, I am going to insert a bunch of rolls into one of those tables and let's see whether DM S is smart enough to recognize those are new inserts.
Now this is a utility that I created right here. What it does is actually just insert into this order table in the source. Let me put in the password.
Very nice so right now, what it does is actually it keeps on creating a new orders coming in this is actually to simulate real life applications. You'll continue to have new stuff coming into your database is now remember. I never put my database down and migration is actually happening so let's actually look at so well.
Bing or inserting a bunch of rules. Let's actually look at DMS in terms of the migration pipeline, and see whether is smart enough to detect any changes so let me refresh. This page right here.
Very nice did you see so the orders already detect 3 inserts right here right and if I keep refreshing?
It will have more rows created right. It will have detect more instance right here. So 9 already inserted so that means your new transactions are come in or just replicate to Azure. You don't need to put anything down. It will actually keep replicating the new stuff into Azure. Now let's say during the.
When down time off your application and I as a DBA right here. I'm saying that OK now. This looks good. All the data already catch up. I am going to stop my application right here.
Right.
Alright I'm stopping all the new transactions that come in. I will wait for D, Ms to catch up to the last batch of data. I will keep refreshing so looks like 15 tables or 15 rolls already get inserted now do you think that I need to watch out for? Is this pending changes. I need to make sure is actually stays at zero.
Means new no new updates has been coming in.
Alright so 17 already get inserted so, so far, so good so this stay at zero. So nothing is actually coming in I can do a cutover so cut over means that I am stopping this migration pipeline. It will not look at any of the pin lock or or regular headlock to actually detect new changes because I say that the migration.
So I am cutting this over.
Alright so it will actually stop the migration pipeline over here.
Now, what was the last step that I actually need to do now. I already move all my data from on premise to the cloud. What I need to do is actually cut over my connection string. That means change my connection string of my application to point to lasure so this is actually what I'm going to do now, I for application right here, then I'm going to move it.
Connection string to Azure right, so I what was my target OK, so this is my target connection string.
I'm going to go to the.
Copy.
My connection string.
This is my username and the password.
Now so there, you go actually so the demo actually show you? How do I actually migrate existing application from on premise or a VM to Azure hosted my SQL any questions so far?
On my SQL, Postgres on Maria DB.
Yeah, so do actually tomorrow's session, Raj is actually going to talk a little bit more on a database migration service like I say it's actually a one stop shop for all your migration needs and then the key value prop right there is that you are not put in your application into a Watt 48 hours of downtime.
In order for your for your data to migrate to the cloud right if you any any question any last minute question from the cloud.
Yes.
What is the overhead?
Right so the question is what is the overhead is it going to text my source database when actually do the migration? When I do the migration right, so we actually tested that in in our benchmarking and also from our customers that tell us the there is actually very minimum overhead so again the.
Look for my SQL or regular headlock is a native built engine right so actually just write the changes into the log. What DM S is doing is actually read into the log right so it will not actually keep locking or right into the it's not going to write into the file so that means it's actually very lightweight operation.
There's actually hardly any text or any locking that happen into while we're doing migration.
Correct yeah, so it's actually he only a read operation into the log right? That's actually not. I'm not going to lock your follow up while the migration is actually happening.
Yes.
Now Oracle is a little bit different. So when when actually so we don't have a Oracle database engine offering in Azure right, So what we have is actually a postgres so Oracle and Postgres has the highest compatibility together, so a lot of our customers actually tell us that Hey.
It's actually getting very expensive in terms of their licensing. What is my alternative in terms of open source is Oracle and Postgres has the highest compatibility in terms of DPS equal and also PGP else equal so that means stop rock in terms of schema.
And also the compatibility into the soul procedure. Postgres has the highest compatibility now. Then when it actually comes to migration. It has 2 step process. That means I need to do a schema conversion and also the next step is actually if you want to do online migration. We will same.
I forgot what is the the the log file in Oracle, same thing will look into the log file of Oracle to actually copy the the new transactions over right now, this right now DM S do not have that offering right now, but we are actually offering a private preview so if you actually interested to do a Oracle Migration, you can actually.
So we are actually offer will release the Oracle to Postgres Migration. I think sometimes in the June time.
No, not 8, I think the minimum so the question is what is the minimum version of Oracle that we support in terms of migration so I think the minimum version that we offer. It's a 1010 and above 10 and above all right 5 minutes left.
Any questions.
Alright, thank you very much for your time for coming to this session today, so open source is something that we actually offer in terms of you know, we give you more offering and flexibility of choice and not only SQL or SQL. Manage instance, we actually have an open sore.
Is Microsoft actually venture into like more if you want more options and also flexibility is for your workload?