Dataset additions

MeteoLatvia

I just recived email from the EUMETSAT user heldesk, and here is what they say:

Dear Martins Bergšteins,
Thank you for reaching out to us about this matter.
We have several different data distribution services available publicly to access our data. For more details about this, please see the following page in our website: https://www.eumetsat.int/access-our-data
Based on your comments, the most similar system we offer is the Data Store, which can both work via a WebUI and via APIs, and can be used alongside our API client EUMDAC to optimise data download and customise the results. For more information, we recommend that you have a look at the Data Store knowledge base, which contains detailed instructions about the service and its functionalities: https://eumetsatspace.atlassian.net/wiki/spaces/DSDS/overview
Please note that not all our products are available across all our data distribution services, so we recommend that you always verify the availability of our data products first in the Product Navigator. The entries there will tell you where the data can be obtained.

I'm really not sure what it means, but seems that they have basically scrapped idea of AWS/S3, so unfortunately - if the the mentioned options are not working for you, then it would be impossible for Meteosat-12 data to be added...

Paul

MeteoLatvia
Yes, unfortunately they do not publish their data for free in the public domain it appears:

"To access these data you will need a Licence Agreement. Most licences are granted without charge, however in some circumstances, e.g. for commercial use of the data, a licence fee is charged."

From: https://www.eumetsat.int/data-registration

If you read the licensing + installation vision thread, I envision a very expensive subscription category that could be available through one-time short-term use passes; with the growth of the product, this could eventually include datasets that I'm not only paying to host/process/serve, but also to acquire and rightfully use.

Paul

Paul I envision WSV3 next-gen 2025 becoming so good and well-known that agencies like EUMETSAT will be approaching me and asking for integration with WSV3 themselves. In the meantime, this goal can be achieved working with only public domain datasets.

MeteoLatvia

Paul

EUMETSAT has great collaboration with ECMWF, and if you manage to get ECMWF model working into WSV3, I think we then would be in the game -with product like this, that from first development screenshots looks and feels better than the entire previous version, it will quickly make their attention. It is just awesome to think, that WSV3 will become the best PC weather tracking software (I think everyone can agree with me), that will display wide range of data and allow for some huge customization. The future for WSV3 ir bright.

Paul

MeteoLatvia I fully agree.

K-McPheron

Paul, I wanted to throw out a few datasets that I use. Throw out some thoughts or any further questions.

Coral Reef Watch (CRW Data)

Datasets from CRW:

SST (Sea Surface Temps)
SSA (Sea Surface Anomalies)
[https://psl.noaa.gov/data/gridded/data.noaa.oisst.v2.highres.html](https://)

Adittional GOES Data

Datasets:

GOES Wind Shear Product - Contoured;
[https://tropic.ssec.wisc.edu/real-time/eastpac/winds/wg9wxc.GIF](https://)
GOES High Density Winds;
[https://www.ospo.noaa.gov/Products/atmosphere/hdwinds/index.html](https://)

MeteoLatvia

Paul, I found another resource which may be useful for You when you will get to potentially add ECMWF model. So I found, that ECMWF HRES model data is also distributed via Amazon AWS S3, which I think may be more better option for you.

Here is general information about the dataset: https://registry.opendata.aws/ecmwf-forecasts/

And here is the bucket: https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/

Paul

MeteoLatvia Thanks for sharing this. The entire new server-side processing infrastructure is being built on AWS and using the S3 Open Data registry as default and preferred source for wherever it is available. I am very excited to build the server infrastructure later this year because it is going to be so much more reliable, professional, and robust than the first-generation product.

I was aware ECMWF has started to output more and better data, and I assure I will be eventually incorporating what's available.

Since the ECMWF is so differently structured than the NCEP data, it really does require a fresh start. I will be doing a comprehensive purpose-built embedded model data engine that gets away from "free-form NCEP GRIB2 parameter viewer" and gets closer to "professional-grade weather forecast decision support utility using operational NWP data in unique ways". The flexible yet unfriendly traditional GRIB2 parameter viewer like in the first-generation product can always be a separate mode/layer.

MeteoLatvia

Paul Thank you for explanation! 🙂

I once again wanted to say big thank you for such an amazing work you do, to make this amazing product from scratch and all of the patience that you have, to listen to our all suggestions and implement them, or at least make them in to-do list. I really appriciate it. 🙂

I'm a bit inactive for now, because of a lot of schoolwork, but I follow all news, and if there will be anything I can help - I will. 😉

MeteoLatvia

Paul

Hi! 🙂

Just wanted to share - till beginning of 2026, ECMWF model output will be available fully in GRIB2 format with Index files at native resolution of 9x9 km. It will run 4 times a day (00Z, *06Z, 12Z, *18Z)

*06Z and 18Z will be hibrid runs at only +90H, while 00Z and 12Z will be full +240H runs.

Information: https://www.ecmwf.int/en/about/media-centre/focus/2024/migration-grib2-data-format-implications-users

Paul

MeteoLatvia Excellent news. Thanks for sharing.

MeteoLatvia

Paul Glad to help. Additional information that maybe is not directly relatable to the data itself, but will be in the near future:

There is now a project that was started in 2019 called "RODEO" (https://rodeo-project.eu/). The main goal for this is to make weather data across Europe united and homogenous, but also make it open-source. Till this pretty much of all data from European weather service where closed to public and even when it become more open, it was in many pre-customized data formats, that where not usable for broader public (including software developers). For example - weather radar scan volumes where available at ODIM HDF5 format, which is not really great to work with. With this project all European weather radars will be available in one place in NetCDF format and with all parameters (Reflectivity, Velocity etc.). Moreover - it will contain data about weather warnings in Europe and as mentioned earlier - ECMWF model data in GRIB2 + Index files.

I will keep updating this post, as more information comes along, but I think it is great news for those living in Europe.

MeteoLatvia

@Paul , I just wanted to share, that Estonian weather agency is now sharing the operational data from both of their weather radars in open access format here: https://avaandmed.keskkonnaportaal.ee/dhs/Active/documentList.aspx?ViewId=c2ea43b7-09c4-4afb-b955-aa5862e46c0f (probably good to use Google Translate / page translation).
These are full radar volumes. This is a quick look on what the data will look like once it comes out for rest of the Europe via the "RODEO" project (the one I mentioned in my email to you). Just for test, if it is managable to implement it and preparing for the data format.

Paul

MeteoLatvia Excellent. I am still in static mapping data preparation/rendering code phase and was just today contemplating the division of datafeed licenses based on region, in that context, to enable higher res imagery/elevation mapping over region, so I don't have to factor-in worst case bandwidth costs into the price for global data when people are mostly per region. Maybe 1 or 2 degree global base tiles and then region-specific for deeper, both hires mapping data and realtime datasets. Lots to look at. Very interested in South American and European regions as likely most proximate non-CONUS specialized datafeed offerings, based on availability of public data, demand, and linguistic factors.

MeteoLatvia

Paul
This is so awesome to hear! Hopefully, we are going to receive some teasers soon? 😃
Either way - I would be happy to discuss and help with data for European and surroundings once the time comes. 😉

MeteoLatvia

Hi, @Paul ! 👋
Didn't want to make a new topic, so using this one.
I wanted to take a moment and wish you and your family a joyful Christmas, filled with happiness and heartful moments. 🤗🎄🎅🌟 May this holiday season bring you just enough rest and a source of energy for new achievements. 🙌😊 You have worked and continue to do it so hard, that I really wish to take a breath and enjoy the time. Thank you so much for all you do!

Paul

MeteoLatvia Thank you very much Martin. It has been an intense initial 15 months of development and doubtlessly an even more intense remaining 11.5 months, but I have never been more optimistic about the project. Merry Christmas to you and looking forward to a great 2025.