Overview
Sometimes I’m not sure if it’s a good thing or a bad one, but I have the kind of personality that when I get involved trying to solve a problem, I can’t stop until it’s solved. This works out well as eventually the issue at hand is fixed, but it also has the side effect that I forsake pretty much everything else until I can resolve things. When I can’t find a solution, or there isn’t a good one, it drives me nuts.
And unfortunately, I’m in that irritated mindset now. If you read my post on Mixing Two or More Sounds Together on the Blackberry, you saw the video I posted of a software sound mixer I had written. While the mixer did indeed work, the problem was I had only tested one scenario – where the user has chosen to start mixing all the sounds together at the same time. This use case is no problem.
The Ongoing Problem
The use case that’s an issue (and honestly the whole point of a realtime mixer), is where you start playing one sound, and then mix another one in at an arbitrary time later. The reason why this is an issue is fairly easily explained (and unfortunately not apparent to me until I had written most of the code).
As you know if you’re a Blackberry developer, all user code is written in Java – the BB runs a JVM where all the software runs – that’s how the phone is designed. This creates a layer of abstraction, where the user doesn’t have direct access to the hardware, and where the code runs relatively slow (compared to native instructions). Because of this, RIM provided mappings from the javax package’s Player class to the Blackberry sound hardware.
The Problem of Buffers and Fetching
Player does a number of things – but the core of it retrieves data from a sound stream (a wav file in our case), massages it as necessary, and then loads the sound hardware buffers with the data. The big problem is at the stage when Player retrieves data. It does so in chunks of data – specifically (from my experiments and as other people on the web have reported) in 58000 byte chunks. That means that is takes in a few seconds of sound at a time so the hardware buffers don’t underrun during playback. You can do some tricks with blocking the stream and force it not to load in this full 58000 – and you will immediately hear the effects – choppy audio. The 58000 byte buffer is not to be annoying – it’s necessary for smooth and clean audio.
And this is why we can’t do realtime mixing. We don’t have direct access to the hardware buffers, we only have access to the buffer streams we can feed to Player. And if Player demands a few seconds of audio at each read, we cannot insert a sound into the few seconds that we feed to it after the fact.
An Example of the Problem
I have two sounds. Drums.wav which is a 20 second wav, and flute.wav which is a 5 second wave. I would like to play drums over and over again, and then immediately mix the flute in when the user presses the ‘F’ button.
Player immediately reads in, let’s say 5 seconds of drum.wav. Then at second 3, the user presses the ‘F’ button. Our program can immediately queue in flute.wav into the input stream, but it won’t be played until second 5, since the Player already read in 5 seconds. Even if we do some creative thread blocking, we can never guarantee a real time mix at all times.
The Code (Could Still Be Useful in Certain Applications)
Below I have included all my work on this project. It consists of a “MixerStream”, which is meant to be a replacement to InputStream. You feed an instantiated MixerStream object to Player, and then call the MixerStream “addStream” method to mix other audio files, in real time, into the Player. It does work – but there will be large delays between when the sound is added, to when it is heard, due to the problem described above.
Note – this code is currently hardcoded to only work with one format of PCM stream right now (I put two hardcoded headers of a 44.1khz, 16bit, stereo, and a 44.1khz, 8bit, mono) – in addition to doing a very low tech way of mixing streams together. The method “getMixedByte” is all setup for future functionality (Allowing any format of PCM, doing better mixing, preventing clipping, etc) – but I’m not up to doing any more work with this right now. It only works as a semi-real time mixer, which I don’t think is too useful – the only place where it might be nice is if you’re developing a Blackberry audio-editing application, or something to that effect – where realtime isn’t necessary. If you are doing this – feel free to use the code below – I’ll provide any help I can.
So for now – this project is abandoned. Frustrating, but necessary for now. Hopefully with the new QNX based OS coming out, there will be more support for directly accessing the hardware buffers.
package com.synthdreams;
import java.io.IOException;
import java.io.InputStream;
import java.util.Vector;
import net.rim.device.api.ui.component.Dialog;
// The MixerStream class allows us to add multiple PCM wave streams to mix together in real time
public class MixerStream extends InputStream{
private Vector _streamVector; // All wave streams currently managed by the mixer
int[] _headerArray;
int _headerPos;
int _totalRead;
// WaveStream class is an InputWrapper stream that stores wav audio information
private class WaveStream extends InputStream {
private InputStream _fileStream; // The input stream containing the wav data
private int _sampleRate; // Samplerate of the wave
private int _bitRate; // Bitrate of the wave
private int _channels; // Channels in the wave
public int getSampleRate() { return _sampleRate; }
public int getBitRate() { return _bitRate; }
public int getChannels() { return _channels; }
public WaveStream(InputStream passStream) throws IOException {
byte[] byteArray;
String tempChunkName;
int tempChunkSize;
// Assign the file stream, then read in header info
_fileStream = passStream;
// 4 - ChunkID (RIFF)
_fileStream.read(byteArray = new byte[4]);
if (new String(byteArray).equals("RIFF") == false) {
throw new IOException("Not a valid wave file (ChunkID).");
}
// 4 - ChunkSize
_fileStream.read(byteArray = new byte[4]);
// 4 - Format (WAVE)
_fileStream.read(byteArray = new byte[4]);
if (new String(byteArray).equals("WAVE") == false) {
throw new IOException("Not a valid wave file (Format).");
}
// 4 - SubchunkID (fmt)
_fileStream.read(byteArray = new byte[3]);
if (new String(byteArray).equals("fmt") == false) {
throw new IOException("Not a valid wave file (SubchunkID).");
}
_fileStream.read(byteArray = new byte[1]);
// 4 - Subchunk1Size
_fileStream.read(byteArray = new byte[4]);
tempChunkSize = ((byteArray[3] & 0xff) << 24) | ((byteArray[2] & 0xff) << 16) | ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);
// 2 - AudioFormat(1)
_fileStream.read(byteArray = new byte[2]);
if (byteArray[0] != 1) {
throw new IOException("PCM Compression not supported.");
}
// 2 - NumChannels
_fileStream.read(byteArray = new byte[2]);
_channels = ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);
// 4 - Sample Rate
_fileStream.read(byteArray = new byte[4]);
_sampleRate = ((byteArray[3] & 0xff) << 24) | ((byteArray[2] & 0xff) << 16) | ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);
// 6 - Byte Rate, Block Align
_fileStream.read(byteArray = new byte[6]);
// 2 - Bitrate
_fileStream.read(byteArray = new byte[2]);
_bitRate = ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);
// variable - Read in rest of chunk 1
_fileStream.read(byteArray = new byte[tempChunkSize-16]);
// Burn through unneeded chunks until we get to data
tempChunkName = "";
tempChunkSize = 0;
while (tempChunkName.equals("data") == false) {
// Read in name and size of chunk
_fileStream.read(byteArray = new byte[4]);
tempChunkName = new String(byteArray);
_fileStream.read(byteArray = new byte[4]);
tempChunkSize = ((byteArray[3] & 0xff) << 24) | ((byteArray[2] & 0xff) << 16) | ((byteArray[1] & 0xff) << 8) | (byteArray[0] & 0xff);
// Burn through non-data chunks
if (tempChunkName.equals("data") == false) {
_fileStream.read(byteArray = new byte[tempChunkSize]);
}
}
// End of header
}
public int read() throws IOException {
return _fileStream.read();
}
}
public MixerStream() {
_headerPos = 0;
_totalRead = 0;
// A constructed wav header for a 44100hz, 16bit, stereo wav of maximum length
/*_headerArray = new byte[] {'R','I','F','F', (byte)0xFF, (byte)0xFF, (byte)0xFF, (byte)0xFF,
'W','A','V','E', 'f','m','t', 0x20, 0x10, 0x00, 0x00, 0x00,
0x01, 0x00, 0x02, 0x00, 0x44, (byte)0xAC, 0x00, 0x00,
0x10, (byte)0xB1, 0x02, 0x00, 0x04, 0x00, 0x10, 0x00,
'd','a','t','a', (byte)0xDB, (byte)0xFF, (byte)0xFF, (byte)0xFF};*/
// A constructed wav header for a 4410hz, 8bit, mono wav of maximum length
_headerArray = new int[] {'R','I','F','F', 0xFF, 0xFF, 0xFF, 0xFF,
'W','A','V','E', 'f','m','t', 0x20, 0x10, 0x00, 0x00, 0x00,
0x01, 0x00, 0x01, 0x00, 0x44, 0xAC, 0x00, 0x00,
0x44, 0xAC, 0x00, 0x00, 0x01, 0x00, 0x08, 0x00,
'd','a','t','a', 0xDB, 0xFF, 0xFF, 0xFF};
_streamVector = new Vector();
}
// MixerStream will first present a wav header (as documented above), then will mix data
// from all added streams. If there aren't any streams, it will block.
public int read() throws IOException {
// Increase the total count of bytes read
_totalRead++;
// First present header
if (_headerPos < _headerArray.length) {
return _headerArray[_headerPos++];
}
else {
// Mix any streams together
if (_streamVector.size() > 0) {
return getMixedByte();
}
else {
// If no streams are available, normally block.
// Return 0 during the prefetch process (58000 bytes) so prefetch
// doesn't block forever
try {
if (_totalRead < 58001) return 0;
synchronized(_streamVector) {
_streamVector.wait();
}
return getMixedByte();
}
catch (Exception e) {
return 0;
}
}
}
}
// This method will mix all the available streams together, keep track of current
// playback byte position to accommodate different PCM formats, and normalize the
// sound mixing.
private int getMixedByte() throws IOException {
int tempRead;
int tempValue;
tempValue = 0;
// Loop through each stream
for (int lcv = 0 ; lcv < _streamVector.size() ; lcv++) {
tempRead = ((WaveStream)_streamVector.elementAt(lcv)).read();
// If we're at the end of the stream, remove it. Otherwise, add it
if (tempRead == -1) {
_streamVector.removeElementAt(lcv--);
}
else {
tempValue += tempRead;
}
}
// Normalize
if (_streamVector.size() > 0) {
tempValue /= _streamVector.size();
return tempValue;
}
else {
return 0;
}
}
// Queue an audio stream to be mixed by MixerStream
public int addStream(InputStream passStream) throws IOException {
WaveStream tempStream;
tempStream = new WaveStream(passStream);
// Make sure this WaveStream is compatible with MixerStream
// Check for sample rates
if (tempStream.getSampleRate() != 11025 && tempStream.getSampleRate() != 22050 && tempStream.getSampleRate() != 44100) {
throw new IOException("Sample rate not supported. (11025, 22050, 44100)");
}
// Check for bitrates
if (tempStream.getBitRate() != 8 && tempStream.getBitRate() != 16) {
throw new IOException("Bit rate not supported. (8, 16)");
}
// Check for channels
if (tempStream.getChannels() != 1 && tempStream.getChannels() != 1) {
throw new IOException("Number of channels not supported. (1, 2)");
}
// Wave Stream checks out, let's add it to the list of streams to mix
_streamVector.addElement(tempStream);
try {
// Notify the read method that there's data now if it's currently blocked
synchronized(_streamVector) {
_streamVector.notify();
}
}
catch (Exception e) {
Dialog.inform("Error notifying _streamVector");
}
return 0;
}
}
