How to handle IMD radar data efficiently?

1. Make sure you have anaconda or miniconda installed or install it using any link given below;
https://docs.conda.io/en/latest/miniconda.html
https://www.anaconda.com/products/individual

2. Install the required libraries/packages listed below;
required: notebook, pyart, xarray, dask, pyscancf, git
optional: wradlib

How to install these packages?
1. Open conda command propmt or terminal after installing anaconda/miniconda
2. type the following command
    conda install -c conda-forge xarray arm_pyart dask git notebook wradlib 
For pyscancf, please use: 
    pip install git+https://github.com/syedhamidali/PyScanCf.git 
Please cite if you use pyscancf 
H.A. Syed, I. Sayyed, M.C.R. Kalapureddy, & K.K. Grandhi. (2021). 
PyScanCf – The library for single sweep datasets of IMD weather radars.
Zenodo. https://doi.org/10.5281/zenodo.5574160

Let’s import these installed libraries,

import xarray as xr
import numpy as np

import glob #used to load the data using glob into this notebook

files = sorted(glob.glob("../MUM200829IMD/*"))

We are interested in 250 km range radar scans i.e short range ppi, so we will select the files having 500 km range and move them into another directory

So first I created an empty list “file500” in which I appended the file names of 500 km range scans

file500 = []
for file in files:
    ds = xr.open_dataset(file)
    if ds.unambigRange > 300:
        print(file, " : ", len(ds.radial.values),len(ds.bin.values),ds.unambigRange.values)
        file500.append(file)

pwd

'/Users/rizvi/Downloads/Projects/jbook/IMD_data/imd'

import os
import shutil

os.mkdir('/Users/rizvi/Downloads/IMD500')

fmv = []
for file in file500:
#     print(file.split("/")[-1])
    fmv.append(file.split("/")[-1])

fmv

[]

import shutil

source_folder = r"../MUM200829IMD/"
destination_folder = r"../IMD500/"
files_to_move = fmv

# iterate files
for file in files_to_move:
    # construct full file path
    source = source_folder + file
    destination = destination_folder + file
    # move file
    shutil.move(source, destination)
    print('Moved:', file)

Now we have only short range ppis in the “MUM200829IMD” directory and we will convert these files to cfradial files by using pyscancf

import pyscancf.pyscancf as pcf

Importing Libraries

## You are using the Python ARM Radar Toolkit (Py-ART), an open source
## library for working with weather radar data. Py-ART is partly
## supported by the U.S. Department of Energy as part of the Atmospheric
## Radiation Measurement (ARM) Climate Research Facility, an Office of
## Science user facility.
##
## If you use this software to prepare a publication, please cite:
##
##     JJ Helmus and SM Collis, JORS 2016, doi: 10.5334/jors.119

<frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead

Importing Libraries Done

pcf.cfrad("../MUM200829IMD/",output_dir='../akki/',)

Number of files:  0
Total number of files will be created:  0
Merging all scans in one file
Data merging done 
Total Time Elapsed:  0:00:04.275578

Now that cfradial files are being created, we shall convert those files further to gridded datasets

import pyart

cf_files = sorted(glob.glob("pol*nc")) # loading cfradial data

 # creating directory for gridded files, which we are going to create by converting cfradial or polar data
os.mkdir("/Users/rizvi/Downloads/outgrid")

output_grid_folder = "/Users/rizvi/Downloads/outgrid/"

for file in cf_files[0:10]:
    radar = pyart.io.read_cfradial(file)
    grid = pyart.map.grid_from_radars(radar,(10,400,400),
                   ((0.,5000.),(-248000.,248000.),(-248000.,248000.)), 
                                   weighting_function='Barnes2',
                                  fields=['REF'],)
    pyart.io.write_grid(output_grid_folder+"grid_"+file.split("_")[-1],grid=grid)

test output grid files

rad_ds = xr.open_mfdataset(output_grid_folder+"/*")

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Input In [18], in <cell line: 1>()
----> 1 rad_ds = xr.open_mfdataset(output_grid_folder+"/*")

File ~/miniconda3/envs/syed/lib/python3.10/site-packages/xarray/backends/api.py:873, in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, data_vars, coords, combine, parallel, join, attrs_file, combine_attrs, **kwargs)
    870     paths = [os.fspath(p) if isinstance(p, os.PathLike) else p for p in paths]
    872 if not paths:
--> 873     raise OSError("no files to open")
    875 if combine == "nested":
    876     if isinstance(concat_dim, (str, DataArray)) or concat_dim is None:

OSError: no files to open

rad_ds['REF'].where(rad_ds.REF<100).mean("time").isel(z=2).plot()

<matplotlib.collections.QuadMesh at 0x154a29910>

rad_ds['REF'].where(rad_ds.REF<100).mean(["x","y"]).isel(z=2).plot()

[<matplotlib.lines.Line2D at 0x154c77d90>]