I cannot find any other tool that encodes AC3 to IEC958 than vlc and mplayer. Both are not suitable for the task for different reasons. One idea would be to modify the alsaplay (aplay) ALSA example application to do the encoding. But while researching the topic I asked myself: why couldn't just ALSA do it by itself? It's just wrapping AC3 into a bit stream, and the mplayer code for it looks fairly simple. Let's see how ALSA plugins are working anyway... So I downloaded the alsa-plugins source code and... WAIT, WHAT IS THAT?! There is a directory "a52". It contains exactly the module I was looking for. It does not encode an AC3 stream to an IEC958 transport stream, it creates the whole transport-encoded AC3 data from PCM data and writes it to the SPDIF device of the sound card. That's even more I asked for!
So I compiled it and added the following to ~/.asoundrc:
and changed the slave PCM of stereo21 to:
pcm "ac3" # was iec958mux
And... It did not work. Not out of the box.
Long story short and a debugging session later: current GIT versions of libav / ffmpeg have changed the default codec for AC3 encoding to floating point format, but the ALSA a52 plugin supports only the signed integer format. You need to get the current source code from GIT for alsa-plugins for a working a52 plugin.
Very sweet, that works, without any race conditions, yet one drawback: the ALSA "dmix" software mixer works only on hardware devices. Of course you cannot dmix an AC3 stream, thus only one application at a time will be able to play sound on this device. How annoying. Especially when knotify starts a kdeinit4 process that keeps blocking the sound interface.