To a society that depends heavily on renewable energy production, weather forecasts have an increasing economic value. Therefore, the Danish Meteorological Institute (DMI) has taken to supercomputing to improve the accuracy and timeliness of its predictions further. The Danish e-Infrastructure Consortium, DeiC – the national research and education network (NREN) of Denmark – has facilitated the efforts.
With more than half of Denmark’s power production coming from wind turbines and photovoltaic cells, knowing future weather conditions translates directly into money saved. This is because backup solutions like gas turbines etc. can be spared whenever the supply of renewable energy can be predicted to be high and reliable.
For DMI, the path towards faster weather predictions involves venturing into an entirely new type of computation supported by artificial intelligence (AI).
“We aim to run these AI models alongside our traditional ones for now. Time will tell what the future of weather forecasting looks like, but these models have the potential to be competitive with numerical models, offering much faster and hopefully equally accurate predictions, or even surpassing the accuracy of traditional methods,” says Simon Kamuk Christiansen, machine learning specialist at DMI.
A workshop started the efforts
Numerical weather prediction relies on solving complex differential equations that describe atmospheric physics, such as the flow of water and air. In recent years, emerging machine learning models for weather prediction have rapidly become competitive with traditional numerical methods. While traditional numerical weather prediction can be handled by CPUs (central processing units), training machine learning models require the use of GPUs (graphics processing units), which can very efficiently and simultaneously process thousands of small calculations such as those required for training a neural network.
Investigation of these emerging models requires both access to very powerful GPU computing and competence within machine learning algorithms. DeiC is host to EuroCC Denmark – the national chapter of the European Competence Center. Scientists from DMI took part in an AI training workshop organized by EuroCC Denmark and were introduced to the possibilities for doing computing at the LUMI supercomputer. Operated by CSC, the NREN of Finland, LUMI is one of the most powerful supercomputers in Europe, equipped with GPUs.
“The DeiC support team found us access to GPU hours on LUMI, giving us 5,000 GPU hours through the DeiC sandbox, enough to get started on LUMI, and start training a first algorithm, and testing how to scale up on multiple GPUs,” explains Irene Livia Kruse, research scientist at DMI.
Adapting models to local needs
Examples of AI-driven models of interest to the DMI researchers are LDCast, which is trained on radar data to predict rain, and SHADECast which is trained on satellite images to predict solar radiation. These models are based on advanced tools developed by other institutions, but DMI is now adapting them and retraining them using Danish data to meet local needs.
At the workshop organized by EuroCC Denmark, the DMI team improved their workflows and scaled their training from single to multiple GPUs, making the process faster and more efficient. For example, by processing more data at a time – doubling the batch size – they significantly reduced the training time per epoch, saving both time and energy.
“Before connecting with EuroCC, our access to GPU-based high-performance computing was limited, which posed significant challenges in advancing our weather models. The support we received – from gaining access to LUMI’s powerful GPUs to the technical guidance at the hackathon – has been instrumental in enabling and optimizing our AI-driven workflows. ,” says Eleni Briola, machine learning specialist at DMI.
DMI is now training AI-based weather models that can deliver faster predictions for both short-term and long-term forecasting.
The text is inspired by the article “Weather” or not to use GPUs – improving weather forecasts with AI-dedicated HPC” by Anne Rahbek-Damm at the DeiC website.
Contributors
