I play FO4 on 2880×2160 with an 1080ti with Framerates beetween 22 and 48 (G3D). The game becomes laggy of corse in certain areas with high details (22 fps), but mainly runs fluend. What you have to do is finetune your settings, i personally lowered shadows and other advanced techniques in favor of details and resolution. There is a manual somwhere on the web describing the settings in detail and how much a change will affect your game.
I dont think a newer card will bring much more performance, make shure you run latest vorpx and on the G3D settings page choose “NEW” sync type which should bring about 20% more performance on FO4.
Please keep in mind that your cpu and mainboard may also have effect on performance.