I’ve heard some folks say its based on the Yamaneko engine, but I can find virtually nothing about that. It looks suspiciously like the Phyre Engine, which was used for Legend of Heroes Trails of Cold Steel 2 (I can’t confirm which engine was used for 3). Nihon Falcom developed both games at the same time, so it’s possible (not necessarily likely) that they used the same engine.
Anyway, if I owned Ys 8 and wanted to test out an engine, I’d try Phyre for it, there are a few official profiles for games that use Phyre on RJkole’s site (here! for speed).