While contemplating the nature of Replay Files being a record of changes to the session and how navigating the timeline really only goes forward, I stumbled upon a bit of an idea. Or two. Hear me out then shoot me down :)
1. Reference Points.
Kind of like the red markers you can place on the timeline for easier navigation. But when you create one of these points Replay saves out a snapshot of the full status of the game at that point - in such a way that it can be used as a *start point* for Replay to playback from.
Depending on the memory and processor required, would it be possible to hold one (or more) of the reference points (from 1 above) in a buffer that gets automatically updated every minute or two, so if you need to go backwards 5 seconds, Replay only has to start from a few minutes ago.
Timeline Navigation is probably my biggest pet peeve with RM, it makes editing 5x slower, a change like this would be well worth the time it'd take to develop imo, if possible.
Assuming that reference points are possible, an alternative is to just have reference points created every second, minute, a certain interval based on how resource consuming these points are. If they are easy enough to make, every keyframe on the timeline could just automatically be made into a reference point.
I've also wondered why moving the cursor back a couple seconds couldn't just result in the rollback of all changes, basically play events backwards?
Reference points unfortunately aren't a concept Minecraft has, so we'd have to implement it ourselves. That by itself already requires unreasonable amount of work and becomes virtually impossible once you add third-party mods.
As for jumping backwards: The RM currently works by saving events/messages/packets sent from the server to the client and then, at a later time, "simply" pretends to be a server and replays those packets to the client. Once example of such an event is PacketEntityDestroy which contains the id of an entity which is to be removed from the world.
So, now we send that packet to the client and pause the replay afterwards to jump back in time. Now, how do we roll back that packet?
Just from looking at it we don't even know which kind of entity it was, ignoring its position and other data for now. To find out we'd have to traverse all the packets until we find the corresponding PacketSpawnEntity and then look for all the movement packets. At that point we haven't really won anything compared to what the RM is doing right now except that we would have lots more code which can contain bugs and has to be maintained.
Having said that, a few weeks ago I've tinkered with a replay mode which only tracks blocks changes and entity movement (and a few smaller things such as time of day and rain) and ignores everything else.
Only tracking those two things, I can analyse the whole replay in advance and then am able to jump between any two points in time (forwards and backwards) within generally less than a second and it didn't require hundrets of hours to implement.
While it doesn't show all stuff, my hope (I don't actually use RM myself) was that it's sufficient for getting a rought overview of things and creating camera paths.
IIRC it was in a somewhat functioning state. I'll see if I can get it polished and backported to versions prior to 1.12.2 within the next week and post a jar for you to try in here once I'm done.
Here's a ReplayMod version which includes this Quick Mode feature: https://minio.johni0702.de/replaymod/replaymod-1.12.2-2.1.4-8-gef36825.jar
You can change the Minecraft version in the URL to any of the supported ones. The Quick Mode will only be available on 1.9.4+ though due to the additional development effort being required to port it to 1.8.9 or older not being worth it (at least for now).
You can enable it while in a replay by pressing Q (iirc) or clicking on the arrow in the bottom left of the screen and then selecting "Quick Mode".
The inital setup of the Quick Mode might take some time, any subsequent loading should be faster (similar to the entity position tracker). Also be warned that it'll require quite a bit of RAM, don't expect to load a multiple hour minigame replay with less than 4GBs of it.
Also make sure to disable the Quick Mode before rendering. That doesn't yet happen automatically.