The thing broke again. 
I know how to solder and all, but with this thing one has to work with sizes as small as a quarter of millimiter. I guess I didn't do a perfect job. It still works, but I have to press on a hacked up bundle of wires, or put it under something heavy enough (eg. map, book) so the it's tight, then it works. Otherwise half of the side doesn't work, i.e. only left or right depending how RCA cable is plugged in.
Anyway, in the meantime I ordered another cheap adapter. I am now determined to really take it apart properly and make me a bulletin proof, army-strong adapter.