Reading UTF-8 Characters From An Infinite Byte Stream
Join the DZone community and get the full member experience.
Join For FreeI’ve been playing with the twitter streaming API today. In very simple terms, you make an HTTP request and then sit on the response stream reading objects off it. The stream is a stream of UTF-8 characters and each object is a JSON encoded data structure terminated by \r\n. Simple I thought, I’ll just create a StreamReader and set up a while loop on its Read method. Here’s my first attempt …
using(var reader = new StreamReader(stream, Encoding.UTF8)) { var messageBuilder = new StringBuilder(); var nextChar = 'x'; while (reader.Peek() >= 0) { nextChar = (char)reader.Read() messageBuilder.Append(nextChar); if (nextChar == '\r') { ProcessBuffer(messageBuilder.ToString()); messageBuilder.Clear(); } } }
Unfortunately it didn’t work. The StreamReader maintains a small internal buffer so I wouldn’t see the \r\n combination that marked the end of a new tweet until the next tweet came along and flushed the buffer.
OK, so let’s just read each byte from the stream and convert them one-by-one into UTF-8 characters. This works fine when your tweets are all in English, but UTF-8 can have multi-byte characters; any Japanese tweets I tried to read failed.
Thanks to ‘Richard’ on Stack Overflow the answer turned out to be the Decoder class. It buffers the bytes of incomplete UTF-8 characters, allowing you to keep stacking up bytes until they are complete. Here’s revised example that works great with Japanese tweets:
int byteAsInt = 0; var messageBuilder = new StringBuilder(); var decoder = Encoding.UTF8.GetDecoder(); var nextChar = new char[1]; while ((byteAsInt = stream.ReadByte()) != -1) { var charCount = decoder.GetChars(new[] {(byte) byteAsInt}, 0, 1, nextChar, 0); if(charCount == 0) continue; Console.Write(nextChar[0]); messageBuilder.Append(nextChar); if (nextChar[0] == '\r') { ProcessBuffer(messageBuilder.ToString()); messageBuilder.Clear(); } }
Published at DZone with permission of Mike Hadlow, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Trending
-
Authorization: Get It Done Right, Get It Done Early
-
Introduction To Git
-
Never Use Credentials in a CI/CD Pipeline Again
-
How To Integrate Microsoft Team With Cypress Cloud
Comments