Record Response?

In a previous article we detailed our work extracting the comms which until now was limited to monochrome PC videos (which you can read here). We had not given up on extracting color comms from the PSX version of the game, but we had turned it into a background task given the limited resolution and framerate. Additionally we were concerned about missing videos, as the PSX version cut some missions and it was less than certain that we could achieve good results combining the two sources.

Fortunately for us the Wing Commander community is always full of surprises.

AllTinker is a fellow modder with his own exciting project, Confederation, which is currently aiming to recreate Wing Commander 1 & 2 for modern machines and adding widescreen support. Since he’d also been looking into WC3/4 formats, we asked if he’d looked at the PSX video. As it turns out he hadn’t, but he very kindly took a detour for us; not just extracting the comms but also combining the PC and PSX video for the best of both worlds. I’ll let him give you the details in his own words:

Decoding The Data

AllTinker: It was known previously that the PS1 VDU data (and music) was stored in a file called “PSX.STR”. The first thing I looked into was how the data was actually stored on the CD; each “sector” is 2352 bytes; 2048 bytes of which contains the actual data you see normally on a PC, while the rest contains metadata. The PS1 can have sectors dedicated to video and audio in their own special hardware-friendly formats; but instead all of the sectors were marked as plain old data. This suggested two things: The PS1’s MDEC video decoding chip wasn’t being used (corroborated by testing in an emulator), and that because of this, the data probably wasn’t (very) compressed; the console’s CPU is almost certainly too busy to be decompressing video into VRAM on the fly.

With that done, I started looking at PSX.STR, and the first obvious thing was that the file is almost certainly indexed externally. A quick look for that index didn’t yield anything helpful, so I moved on to trying to find video data manually. This was quite easy and had been done before – some video frames are obvious even in the hex editor! Once I’d found a good first frame, I started trying to puzzle out the header before it. Thankfully it didn’t take long to find some good data – the number of frames and the resolution – and slightly later in the header was a fairly obvious chunk of 256 2-byte values. Looks like a colour palette to me!

I wrote some software to start pulling out the data, and a quick bit of experimentation revealed that the palette entries were 15-bit BGR colours, and that the frames themselves appeared to be uncompressed. Now I could extract some frames…

You can see in my first attempt that there’s some empty space at fairly regular intervals… Perhaps this was for interleaved audio chunks, which they never ended up using. After I figured out the interval where these gaps occurred, it was also obvious that the next frame began on the next sector boundary (i.e. the next multiple of 2048 when viewing PSX.STR as a regular file).

After this success I moved on to extracting all of the videos; however, it turned out that around 20% were missing, and these seemed to be predominantly videos which cut between multiple scenes. It seemed logical that there was probably a palette switch when this happens, and eventually – once I located some of these scene-switching videos – I figured out more about the header. Each header contains TWO palettes, and each palette declares a set of (up to 8) starting frames from whence that palette should be used. Now that I understood much more about the header I could improve my code and extract the rest of the videos!

The Best of Border Worlds

Now to state the obvious… These videos are hilariously small (60 x 48!); less than a quarter of the pixels of the “high-resolution” PC greyscale VDU videos (a whopping 128 x 108) – so the question naturally came up whether we could use these colour videos as a chroma channel for the PC ones. Unfortunately there were two main issues standing in the way: 1) The colour videos aren’t just smaller and half the frame-rate; they’re also cropped. 2) None of the colour videos were named; a problem which had to be solved one way or another. Thankfully for 1), after some investigation AD figured out that the cropping appeared consistent. I experimented a little from there with some ImageMagick scripts, eventually settling on a dithering + blurring method to pad out the missing colour information, and some simple interpolation to match the frame-rate; it works pretty seamlessly in most cases.

For 2) I decided, as I often do, that the answer was more code. I wrote a program to try to match the colour videos to the greyscale ones; this worked well enough to save most of the manual labour of manually matching nearly 300 videos… A whole lot of crunching later and we have a set of colourised videos to pass to ODVS for AI upscaling!

Not bad, for a dinosaur

With what was frankly a stunning combination between some ridiculously small monochrome videos and some ludicrously small color videos we could have called it a day. This patchwork of data was not ideal for AI upscaling to say the least.

But ODVS always proves that where there is a will there is a way:

ODVS: There was ludicrously little pixel data for Topaz Video AI to work with – when you’re dealing with such tiny dimensions, faces usually get lost in a blur of pixels. However, Topaz Gigapixel AI has an extra facial recovery algorithm that can work wonders. By using Gigapixel to upscale every frame as an individual image and then recombining those frames back into video files, we were able to create far more visual data than was available in the source files.

Unfortunately, a large proportion of the comms videos take place in fighter cockpits, where pilots are wearing helmets. With their helmets on, only the actors’ eyes are visible – and that’s not enough visual cue data for the algorithm to recognise that a face is even present! The results were… less than ideal – with lots of artefacts and corruption to the upscaled image. It was obvious we still needed a way to retain the illusion of more fidelity while masking out the visual corruption.

Fortunately for us, in trying to replicate the 90’s era FMV we had an obvious hack; all the monitors were CRT. Quite why that technology made a revival in the 27th century is not ours to say – but we can be thankful that they were capable of hiding a multitude of sins.

Using Adobe After Effects, I split the comm videos into their three component colour channels (red, green and blue), then created a grid of RGB “phosphor dots” – precisely the way old CRT screens used to work! Using this grid as a mask, I revealed each colour channel only through the associated dots on the grid – which both mimicked the visual style of a CRT TV and masked the vast majority of our visual issues.

My promotion came through!

These comms proved to be a good chance to finally push out a new demo. While the original WCIV demo did not support video comms, we have brute-forced in some pirate comms so you can check them out and provide feedback. As the original demo lacked the comms you may find the audio/video sync seems off (as these videos were never meant to be combined with this audio).

Is the CRT effect too much? Is there anyone desperate for a monochrome option (outside perhaps classic mode which will definitely use it)?

Give it a spin and let us know.

Here is the list of fixes:

• Prioritize GPU based on maker if multiple are found
• Support for 128 button joysticks
• Support for joysticks/throttles with zero POV hats
• Fix for taking screenshots with AMD cards
• Fix for score tracking when critical components are destroyed
• Reduce memory usage on cards with limited VRAM
• Support for multiple language tracks (playing with -ger will give you a german trailer audio)
• …and color comm videos.

With this release we have decided to move our focus exclusively to the full game. We may put out future updates to resolve compatibility issues / get early feedback about new features, but only if we feel that helps to bring us closer to a stable and well-received full release.