[00:00.000 --> 00:10.800]  I'm David Fioriti from the University of Pisa and co-director of the Pipe Submits Earth
[00:10.800 --> 00:12.200]  Initiative.
[00:12.200 --> 00:16.360]  And today I will talk about open data and open source adoption in the NSG sector and
[00:16.360 --> 00:22.960]  in particular we will give the example of how we are doing the Pipe Submits Earth Initiative
[00:22.960 --> 00:26.160]  in which we develop energy modeling.
[00:26.160 --> 00:33.080]  So let's start from the business social model in energy planning.
[00:33.080 --> 00:42.440]  I really thank Robby for the fantastic introduction before and we can see that what the big players,
[00:42.440 --> 00:50.440]  most of the big players are doing except maybe few exceptions that also are presenting today
[00:50.440 --> 00:58.840]  is that a lot of entities which means governments or also TSOs and really big players are paying
[00:58.840 --> 01:07.160]  for having access to solvers, models and data to develop their own very specialized model
[01:07.160 --> 01:12.120]  to obtain some results that they are highly interested and they are willing to pay a lot
[01:12.120 --> 01:14.280]  for that activity.
[01:14.280 --> 01:21.440]  But this leads to problems of clearly transparency of results and replicability that we have
[01:21.440 --> 01:25.760]  been talking all about today.
[01:25.760 --> 01:33.360]  But to just draft a bit of what we mean is that and to clarify about the duplication
[01:33.360 --> 01:39.040]  that is occurring is that let's say that three entities are actually working on developing
[01:39.040 --> 01:42.480]  different activities very narrow focused.
[01:42.480 --> 01:49.360]  The entity A may have some features as likewise the entity B but different geographical scope
[01:49.360 --> 01:54.760]  let's say A is interested in Germany and the B is interested in Italy.
[01:54.760 --> 02:03.560]  On the other side the entity C that may develop something similar is interested into another
[02:03.560 --> 02:09.240]  feature to add for example stability features or so on but still working on Germany.
[02:09.240 --> 02:15.360]  This however leads to duplication of features because entity A and B needs to develop twice
[02:15.360 --> 02:24.240]  the same feature and entity C instead has to work twice to filter the data that are occurring
[02:24.240 --> 02:30.200]  and this is an issue and we are going to talk about this.
[02:30.200 --> 02:35.080]  In the open-air approach in energy planning is that the idea is to open up the different
[02:35.080 --> 02:43.240]  tasks and with open source tools that Robbie mentioned about and so this can be avoided
[02:43.240 --> 02:49.480]  because the parts that are needed the data handling and the efficient they can be simply
[02:49.480 --> 02:53.480]  taken up and appropriately used.
[02:53.480 --> 03:00.400]  In a short way let's not reinvent the wheel let's say and awareness of this is rising
[03:00.400 --> 03:06.760]  as we all know and especially the industry level it's interesting to see here that the
[03:06.760 --> 03:13.080]  organization of DSOs at the European level is pushing towards opening and using open
[03:13.080 --> 03:18.840]  source in practice however reality is a bit different.
[03:18.840 --> 03:25.160]  We have seen Alieander that is doing a great job here but at least an Italian level I tell
[03:25.160 --> 03:32.640]  you that this is taking a while and I'm sure that this is kind of commonplace.
[03:32.640 --> 03:40.520]  In fact the reality far from true complete here we can see data and the access to data
[03:40.520 --> 03:49.120]  is an issue that is needed to use successfully open and fully transparent activities and
[03:49.120 --> 03:55.480]  you can see here that in terms of public available data on final energy there is none.
[03:55.480 --> 04:03.840]  IEA is a lot of data and recently has pushed towards releasing openly the data but currently
[04:03.840 --> 04:09.600]  they are missing the funds to do so in fact they asked the national governments to come
[04:09.600 --> 04:13.000]  to feed the gap.
[04:13.000 --> 04:21.400]  So open tools are great but users may concern in using them because of quality and security
[04:21.400 --> 04:28.080]  issues that have been discussed and developers they have some concerns as well.
[04:28.080 --> 04:35.600]  However we can solve that and there is the need to improve on these together so to quickly
[04:35.600 --> 04:41.480]  and go beyond these barriers apparent barriers.
[04:41.960 --> 04:48.360]  On one side we need users to coordinate the developments so to not waste the scarce resources
[04:48.360 --> 04:54.920]  and answer the quality issues and long term sustainability that we are facing and on
[04:54.920 --> 05:00.200]  the other end developers with appropriate license that can be protected and can still
[05:00.200 --> 05:04.680]  thrive with no problems.
[05:04.680 --> 05:09.960]  So we talk about open data and what about the numbers.
[05:09.960 --> 05:16.800]  In 2019 in a recent review it has been they were reviewed about 30 energy models worldwide
[05:16.800 --> 05:24.160]  and thanks to open mode initiative we see that these numbers are growing and like last week
[05:24.160 --> 05:32.840]  the numbers was 73 and this is great but some concerns arise is there some failed cooperation
[05:32.840 --> 05:34.480]  in all of this.
[05:34.480 --> 05:40.680]  How much duplication is there in these numbers I'm not speaking about that I'm just posing
[05:40.680 --> 05:48.240]  the problem now and that's why I think that with initiative there is the need to provide
[05:48.240 --> 05:54.200]  guidance as they are actually doing and better coordination to work together rather than
[05:54.200 --> 06:01.760]  duplicate and activities and also to show the possible users that the actual model is
[06:01.800 --> 06:04.000]  already there.
[06:04.000 --> 06:10.880]  In particular I took the screenshot of the overview of the models and we can see that
[06:10.880 --> 06:16.720]  by columns it is possible to focus on some features that are interesting and I'd like
[06:16.720 --> 06:22.080]  to notice that I think it could be nice also to show some application based recommendations
[06:22.080 --> 06:28.760]  so let's say what if I want to develop an energy management system what are the options
[06:28.760 --> 06:31.960]  and so on.
[06:31.960 --> 06:35.560]  So now let's go at they use for energy planning.
[06:35.560 --> 06:40.480]  So these tools have been used for different scopes and they have different features and
[06:40.480 --> 06:46.440]  in particular in this case we have recently summarized some of the features that the different
[06:46.440 --> 06:54.560]  tools are providing and in particular we focus on energy planning used in Africa.
[06:54.560 --> 07:03.560]  Basically the state of the art of energy planning in Africa is done by the use of Plexus.
[07:03.560 --> 07:10.760]  Plexus is a commercial tool and that has a lot of features and it is actually very trusted
[07:10.760 --> 07:17.320]  by the industry and in fact very widely used but as we can see there are plenty of other
[07:17.320 --> 07:22.480]  open source alternatives that can be competing in terms of features with respect to what
[07:22.480 --> 07:30.800]  Plexus is offering and but why are not we using the open source that's a big question.
[07:30.800 --> 07:42.600]  So and that's why we need to build trust and to work together to thrive on this and the
[07:42.600 --> 07:47.800]  Peter said that I want to show here and also recall later when we go deeper into the pipes
[07:47.800 --> 07:53.280]  and its initiative is that we need to work together and then duplicate efforts and also
[07:53.280 --> 08:01.120]  share knowledge and data because by intertwining and sharing them we can only achieve this
[08:01.120 --> 08:02.120]  goal.
[08:02.120 --> 08:08.800]  So now let's go into deeper and how we do that at the our initiative in particular what
[08:08.800 --> 08:14.800]  is pipes and it's earth but it's earth is an open independent research initiative that
[08:14.800 --> 08:22.360]  aims at pushing at the global speed up the global energy transition by using open data
[08:22.360 --> 08:24.600]  in open source tools.
[08:24.600 --> 08:33.360]  In particular we have no barrier in terms of a peep of preconcepts we are welcoming anybody
[08:33.360 --> 08:39.920]  any of you of any friends that you may have to join and collaborate with us and if you
[08:39.920 --> 08:49.160]  ask okay but what are you working on so here you can see our main four pillars and I want
[08:49.160 --> 08:54.240]  none ever to strengthen enough the concept of community in fact this is the first one
[08:54.240 --> 09:00.600]  and we have the first concept and this is very important we have a discord channel and currently
[09:00.600 --> 09:09.880]  we are over 200 people in this discord channel well above and that's not like 201 it's like
[09:09.880 --> 09:20.640]  about 250 something like that number 300 already okay fantastic and they changed it pretty fast
[09:20.640 --> 09:28.480]  because I checked last week like fantastic and we push for open data because we believe
[09:28.480 --> 09:34.680]  and that open data is the only way and not only that we also push for open models and
[09:34.680 --> 09:39.800]  but that's not that's not a novelty but it's something that is also interesting to show
[09:39.800 --> 09:46.360]  is the concept of solver on the right because having open data and open models is not enough
[09:46.360 --> 09:52.960]  if you want to produce the results you need a solver that is open to possibly use in fact
[09:52.960 --> 10:00.400]  the initiative we have also showed that and improve the visibility of other tools from this
[10:00.400 --> 10:09.400]  point of view something that we know we is I think one of the novelties here is that when the
[10:09.400 --> 10:14.320]  project started and I have to be honest when the project started I was not there I joined as
[10:15.080 --> 10:23.880]  others as well is that we grew an existing user base so we didn't start from new modeling and
[10:23.880 --> 10:31.640]  with nothing else we started from the PIPSA user base that was already large and established with
[10:31.640 --> 10:40.440]  a lot of tools for different elements in particular you can see here PIPSA that is a framework we
[10:40.440 --> 10:47.720]  call it that that enables to draw to draw easily equations automatically so that the user does
[10:47.720 --> 10:55.080]  not need to do that if you want to add for example a storage in a in a bus it does it for you while
[10:55.080 --> 11:00.120]  instead we have also them there are the also the other models PIPSA Europe and PIPSA Europe
[11:00.120 --> 11:09.440]  sec that are the frameworks populated with data to represent the Europe system what we did is to
[11:10.000 --> 11:17.000]  leverage on this knowledge and this expertise and draw the earth modeling and that you can see here in
[11:17.000 --> 11:24.240]  particular we are working on several packages the earth model that aims at modeling the earth and
[11:24.240 --> 11:31.400]  other two models is linked for data creation that I will talk to you about later briefly and you may
[11:31.400 --> 11:43.720]  ask why PIPSA but the answer is here and it's quite popular and thanks to that our community is
[11:43.720 --> 11:49.520]  pretty large and I really like this image with a lot of faces and I think that we should update
[11:49.520 --> 12:00.120]  these numbers hopefully not adding all of them but the major ones so if I you if it to summarize
[12:00.360 --> 12:07.480]  our recipe in a single slide I'd like to show this here so first we start from growing an existing
[12:07.480 --> 12:15.720]  user base not creating a new one secondly by create by leveraging a user base there is an
[12:15.720 --> 12:24.000]  existing model and what we want to gather in our procedure is to cut all the possible contributions
[12:24.400 --> 12:31.680]  that other user may do because this can benefit each other in an interesting way in this matrix you
[12:31.680 --> 12:41.520]  can see that if a user other country then another user adding the feature a may benefit to of the
[12:41.520 --> 12:49.000]  existing feature so that they can use the feature a on the existing model plus the contribution of
[12:49.000 --> 12:57.120]  the country see and similarly lack likewise the feature be added by another person may may add on
[12:57.120 --> 13:04.480]  top of each other and this is what actually what we are working on we put to produce open data and
[13:04.480 --> 13:08.880]  open source tools that are then shared with the other communities as well in fact we add also
[13:08.880 --> 13:17.000]  other chats with other communities so to to possibly provide the data that we produce to feed
[13:17.000 --> 13:25.160]  other energy modeling tools for example written in Julia which is not our and then obviously clearly
[13:25.160 --> 13:33.120]  by doing this doing this activity we share knowledge so how to plan for a bright future we
[13:33.120 --> 13:40.720]  need policy tools and analysis and for these reasons we have provided pipes earth that is our
[13:40.800 --> 13:48.720]  measure tool for energy sector model for earth scope and we are working on distribution level
[13:48.720 --> 13:55.040]  approach that is called pipes of distribution but this is highly under development so for the one
[13:55.040 --> 14:03.440]  on the left the package is stable and what about the data so we have a lot there are a lot of open
[14:03.440 --> 14:09.840]  data but especially in some regions that are missing I like this slide in which you can see that on
[14:09.840 --> 14:15.600]  the left they are the open data and on the right you can see that there are a lot of missing ones in
[14:15.600 --> 14:22.800]  their actual data set and that's why there are we are there are some packages to tackle this issue
[14:22.800 --> 14:28.240]  particularly we are relying on we are developing the tech energy that aims to estimate the energy
[14:28.240 --> 14:33.000]  infrastructure that may be missing and on the other hand there is the demand creator that
[14:33.000 --> 14:41.040]  aims to estimate the demand and leverage on existing AI tools to perform this task so to give
[14:41.040 --> 14:47.280]  a very quick recap of three slides for what we mean with the energy modeling and pipes earth
[14:47.280 --> 14:57.240]  first of all what are the functionalities and we can see that to satisfy the need by the
[14:57.240 --> 15:03.560]  policy makers that are robust, reliable, low-cost, simple and planning tools we provide our solution
[15:03.560 --> 15:11.160]  that means to leverage validated models with the community support and those and the open
[15:11.160 --> 15:17.560]  community absolutely and really many thanks to the existing great work that the pipes of
[15:17.560 --> 15:29.880]  community has done and I cannot say it's not it's never enough so these to satisfy these
[15:29.880 --> 15:36.760]  requirements we perform a complete procedure that aims to start from the data analysis in which we
[15:36.760 --> 15:44.080]  open we use and rely open data sets such as open street map and so on and we filter out to produce
[15:44.760 --> 15:53.360]  high resolution data and without a solution we mean that if you want to model your municipality so
[15:53.360 --> 16:01.240]  that if you want the municipality resolution you if you have a computer big enough can be are able
[16:01.240 --> 16:06.440]  to do that for the region that you may be interested to tackle are you interested in Africa are you
[16:06.440 --> 16:16.040]  interested in Nigeria are you interested in Australia that's what you can do and how can you
[16:16.040 --> 16:22.960]  make it easy and the procedure is the following like the general procedure they follow is that we
[16:22.960 --> 16:28.400]  decompose the large problem into small pieces in which every contributor can add these its own
[16:28.560 --> 16:37.960]  few lines of code to actually obtain the result starting from the creation file down to results
[16:37.960 --> 16:49.880]  and how is it easy easy to run two lines when it works obviously first you choose your
[16:50.400 --> 17:01.480]  countries of interest secondly you run this and when it does not work like always you can add see
[17:01.480 --> 17:06.920]  the documentation and also access our discord channel that in which you can interact with us very
[17:06.920 --> 17:13.840]  easily and to summarize and show you some results these are some results that are being published
[17:14.160 --> 17:23.200]  in our preprint and in particular you can see net zero energy planning for Nigeria and Iran for
[17:23.200 --> 17:29.160]  for Africa Nigeria has been validated Africa we need to work a little bit more on the data and
[17:29.160 --> 17:38.160]  what's next earth is next currently I'm working on validating each single country worldwide I've
[17:38.200 --> 17:45.840]  started with Africa South America in Asia and currently we are a status about 60% of those
[17:45.840 --> 17:53.600]  working so I can show you an image also about the current status and to do so that we absolutely
[17:53.600 --> 18:00.200]  need to share knowledge work together and data so thank you very much for being here and I'm
[18:08.560 --> 18:15.000]  up question yeah please
[18:21.000 --> 18:30.680]  okay that's a good question so the question was okay when you run the just two lines what are
[18:30.680 --> 18:41.520]  the outputs actually so in particular I can show so first of all we have also a YouTube channel
[18:41.520 --> 18:47.960]  in which you there are videos in which we run the models with some of the community and you can
[18:47.960 --> 18:57.320]  see everything from start to the end in particular the outputs are a lot of files that are all the
[18:57.320 --> 19:03.680]  intermediate files that produce time series for demand time series for the renewable production as
[19:03.680 --> 19:14.520]  well as the structure that encapsulates the networks that we produce the in terms of software
[19:14.520 --> 19:21.200]  they are basically CSV file and C file so there are different data structures that are all
[19:21.240 --> 19:33.280]  documented and that can produce I can show you also an example of folder so I usually work on a
[19:33.280 --> 19:45.400]  remote computer so here maybe a little bit so I can show you exactly my current folder that I use
[19:45.640 --> 19:55.680]  for developments okay and I can show you also the status of the countries that are actually
[19:55.680 --> 20:06.680]  working and obviously it's improving so this is a folder connections I'm not sure if it's going
[20:07.200 --> 20:15.360]  sorry I was connected but I got this disconnected before
[20:15.360 --> 20:27.600]  and so okay in the meantime I do have a clone here locally maybe
[20:33.120 --> 20:44.440]  okay yes so this is a typical pipe folder you can see that there are different data structure
[20:44.520 --> 20:50.120]  and different folders I don't want to go into the details but because but there are videos that are
[20:50.120 --> 20:55.720]  actually tackling these problems and we are we have actually run the models together but if you
[20:55.720 --> 21:03.080]  go and into the results basically the results are a folder which contains a network that is that
[21:03.080 --> 21:09.160]  contains is a pipes a network and you can open it also with a notebook from Visual Studio code and
[21:09.160 --> 21:14.520]  you can reproduce and see the outputs if in the meantime the folder has loaded
[21:19.560 --> 21:27.240]  in the meantime that's loading maybe okay now it's loading yeah please
[21:27.240 --> 21:51.800]  so currently pipe cert is more at the question was if we can model at municipality scale so
[21:51.800 --> 21:58.040]  basically the a question about the resolution spatial resolution that we are actually addressing
[21:58.040 --> 22:06.040]  so currently pipes earth is working at transmission level also it because we all have data for
[22:06.040 --> 22:12.600]  transmission level because we also rely on open street map while at medium voltage and low voltage
[22:12.600 --> 22:19.640]  lines the data sets are quite poor if any or if I mean there are some few but really few
[22:20.360 --> 22:29.400]  which means that our model is more suited currently for larger areas in particular we work on with
[22:29.400 --> 22:38.120]  administrative zones JDM JDM data set and what we can do is you can choose the level you are
[22:38.120 --> 22:43.800]  interested into and you can run the model for that level of resolution this is possible
[22:44.760 --> 22:50.040]  obviously the data that are used are the data that are available of open street map but
[22:50.040 --> 22:59.320]  if you have high detail the data that it may be closed source you can fit them in without releasing
[22:59.320 --> 23:04.360]  and it is possible to use we are actually working on this and in the meantime it has loaded
[23:05.560 --> 23:10.920]  and we can see that I mean this is my current folder it's a bit there are a lot of stuff
[23:11.640 --> 23:19.160]  but this is because all these because I've actually run a lot of scenarios you can see the
[23:19.160 --> 23:24.840]  number of scenarios that have been run and each of them corresponds to a configuration file for
[23:25.480 --> 23:30.840]  the image that you have seen and the green the light green countries are those that are being
[23:30.840 --> 23:37.320]  successfully executed with the dark green we are almost there and also the others we are actually
[23:37.400 --> 23:44.520]  backfixing them and in the couple of few months we expect to have the entire picture green the white
[23:44.520 --> 23:55.800]  ones have been not executed yet sorry do you say that that would work in matches measure data or something
[23:55.800 --> 24:03.400]  no because the problems here are some are basically data management problems like in
[24:04.280 --> 24:17.160]  in china for example the the columns of the data set from open street map contained in Japan
[24:17.160 --> 24:27.240]  contained for example in frequency and in the tag frequency there was 50 comma 50
[24:27.960 --> 24:33.640]  instead of 50 only so it was it's a really a data management problem that needed to be fixed and
[24:33.640 --> 24:40.520]  that's why now we improved the the representability of Japan and there are some other little bit
[24:40.600 --> 24:42.760]  backfixing that we are currently working
[24:55.800 --> 25:05.640]  okay so the the question more or less is what data we are using in the model so for the net depends
[25:05.720 --> 25:13.080]  on what is the type of information that we need we merge weather information coming for example
[25:13.080 --> 25:21.880]  for era 5 data we can we collect data from gdp and population data for the population data was
[25:22.760 --> 25:28.440]  over pop and so on there are different data if we look at the network data and open street map
[25:28.840 --> 25:36.360]  we as a mainstream data we consider the open street map data as a first producer but we are
[25:36.360 --> 25:45.000]  also working at a possibly streaming your own data because maybe there is another data set that
[25:45.000 --> 25:51.560]  is as better information or than open street map and you may be willing to use it instead of open
[25:51.640 --> 25:59.320]  street map for a single country and we are working on creating this interface to feed in more data
[25:59.960 --> 26:09.320]  it's it should be a few days of work if and also anyone is interested and I think time is up and
[26:09.320 --> 26:11.480]  but for anything you have my contacts and