Sumatra PDF is a PDF, ePub, MOBI, CHM, XPS, DjVu, CBZ, CBR reader for Windows

Activate e-book previews like for PDF

Sumatra has wonderful design for PDF preview. How to make epub file has same capability?

The code has been historically there for some time and can be activated by those that build their own version.
It has traditionally not been included for several reasons. see https://github.com/sumatrapdfreader/sumatrapdf/issues/1248
As far as I know it cannot show DRM files contents, that is perfectly natural for most previewers.
It does not always show some other e-book covers, and does not have the full compliment of associated icons however that could possibly be improved, if someone was to spend time on it.
The code for ePub formats was extracted by a third party File handler. I cannot vouch for its behaviour and if you have problems then report them there not here. see https://www.zabkat.com/blog/epub-ifilter-windows-search.htm