yEnc
yEnc is a binary-to-text encoding scheme for transferring binary files in messages on Usenet or via e-mail. It reduces the overhead over previous US-ASCII-based encoding methods by using an 8-bit Extended ASCII encoding method. yEnc's overhead is often as little as 1–2%, compared to 33%–40% overhead for 6-bit encoding methods like uuencode and Base64.
With decreased overhead, the encoded message body is smaller. Therefore, the message can be delivered faster and requires less storage space.
An additional advantage of yEnc over previous encoding methods, such as uuencode and Base64, is the inclusion of a CRC checksum to verify that the decoded file has been delivered intact.
yEnc was created and released into the public domain by Jürgen Helbing in 2001.
How yEnc works
Usenet and email message bodies were intended to contain only ASCII characters (RFC 822 or RFC 2822). Most competing encodings represent binary files by converting them into printable ASCII characters, because the range of printable ASCII characters is supported by most operating systems. However, since this reduces the available character set considerably, there is significant overhead. For example, in uuencode and Base64, three bytes of data are encoded into four printable ASCII characters, which equals four bytes, a 33% overhead (not including the overhead from headers). yEnc uses one character (one byte) to represent one byte of the file, with a few exceptions.
The RFCs that define Internet messages still require that carriage returns and line feeds have special meaning in a mail message. Therefore, yEnc escapes the carriage return and line feed characters in the encoded body.
There is no RFC or other standards document describing yEnc. The yEnc homepage contains a draft informal specification and a grammar (which contradicts RFC 2822 and RFC 2045), although neither has been submitted to the Internet Engineering Task Force.
As with uuencoding, despite its flaws, yEnc remains active and effective on Usenet. The yEnc homepage states that "all major newsreaders have been extended to yEnc support". Microsoft's Outlook Express and Mozilla Thunderbird are two major newsreaders that do not provide direct yEnc support for either news or mail, but there are plugins available.
Criticisms
 |
The creator of the yEnc encoding scheme and others have criticized the design of yEnc. It suffers from many of the same flaws as uuencode does, a number of which had already been solved years before by MIME (which addressed the same flaws in uuencode). For example, yEnc requires the strings "=ybegin" and "=yend" to be placed around the encoded file in the message body. Although this is an improvement over uuencode's "begin" and "end", which occur more frequently in normal text, message readers can still encounter attachments where those strings are present (most frequently in discussions about yEnc itself). yEnc and uuencode also attempt to reassemble files split into multiple messages by using the subject line, which is unreliable.
Moreover, yEnc adds a few new flaws of its own. It attempts to turn unstructured fields into structured ones, which is unreliable, given that no constraints can be placed upon the unstructured use of the fields by non-yEnc uses. Most notably, the subject line of the message is supposed to contain the string "yEnc", the filename, and the part number. (The yEnc homepage chastises yEnc article posters for themselves not observing these constraints.) MIME places all such information in the message headers, which is far more reliable.
Uuencode was careful to support Internet messages as streams of text, which yEnc does not support. Software that supports yEnc encoding must know the size of the original file in advance, because the file size is specified in the yEnc header that precedes the encoded file.
Not all transports can handle the 8-bit characters employed by yEnc, which may cause data corruption. yEnc can also be mangled by different character sets. It works poorly with the increasingly popular UTF-8 character set, for instance. Moreover, some article transports may, on the grounds of enforcing compliance with the Internet message format standard, automatically convert any message using 8-bit characters to either Base64 or quoted-printable, entirely nullifying the overhead advantage.
Critics also take issue with the lack of formal standardization.
Some people, including yEnc's creator, have suggested including yEnc as part of MIME, which would solve nearly all of its problems and retain the low encoding overhead. However, as of March 2006[update], no formal or informal standard has been reached.
Example Encoder (in C)
// Created by Juergen Helbing in 2001 as a part of the MyNews project
// This is a source code example for the yEncoder - see also: http://www.yenc.org
// It is public domain - as well as the yEnc specification
//
// The progamming style is very "basic" and readable.
//
// The function is writing a binary source file (fSrc) to an encoded target file (fDes).
// The original filename is "postname"
// The size of the sourcefile is "filelen"
// For singlepart messages 'part' 'lastpart' and 'fulllen' are set to zero.
// For multipart:
// Set "part" to 1..lastpart
// "fulllen" is the size of the entire sourcefile
// "filelen" is the size of the part which is actually encoded.
// print() and eprint() are logging displays from 'MyNews'
// Crc32 - functions avbailable at the end of this example
int yEncode(FILE * fDes, char * postname, FILE * fSrc, long filelen,int part,int lastpart,long fulllen)
{
long restlen;
long srclen;
long deslen;
unsigned char srcbuf[4100];
unsigned char desbuf[260];
unsigned char * srcp;
unsigned char * desp;
unsigned char c;
int id;
int linelength;
static long pbegin, pend;
print("yEncode (%ld bytes)\r\n",filelen);
linelength=128; // max 255 - due to desbuf size
if (part==0) // SinglePart message
{
fprintf(fDes,"=ybegin line=%ld size=%ld name=%s\r\n",linelength,filelen,postname);
}
else // Multipart message
{
if (part==1) { pbegin=1; pend=filelen; }
else { pbegin=pend+1; pend=pend+filelen; };
fprintf(fDes,"=ybegin part=%d line=%ld size=%ld name=%s\r\n",part,linelength,fulllen,postname);
fprintf(fDes,"=ypart begin=%ld end=%ld\r\n",pbegin,pend);
}
CrcInit();
deslen=0; desp=desbuf; restlen=filelen;
while (restlen>0)
{
srclen=restlen;
if (srclen>4096) srclen=4096;
id=fread(srcbuf,srclen,1,fSrc);
if (id != 1)
{
eprint("Error in reading file for encoding (code=%d)\r\n",errno);
return(1);
}
restlen=restlen-srclen;
srcp=srcbuf;
while (srclen>0)
{
c=*srcp; // Get a source byte
CrcAdd(c); // Add it to the CRC
c=(unsigned char) (c+42); // and add the secret number
srcp++; srclen--;
switch (c) // Solve special in NNTP 'forbidden' characters
{
case 0:
case 9:
case 10:
case 13:
case '=': // Including the escape character itself
case '.': // Some usual servers have problems with a dot in the first column
*desp='='; desp++; deslen++;
c=(unsigned char)(c+64);
}
*desp=c; desp++; deslen++;
if ((deslen>=linelength)|((srclen==0)&(restlen==0))) // Block full - or end of file
{
*desp=13; desp++; deslen++;
*desp=10; deslen++;
id=fwrite(desbuf,deslen,1,fDes);
if (id!=1)
{
eprint("Error in writing encoded file (code=%d)\r\n",errno);
return(1);
}
deslen=0; desp=desbuf;
}
} // end of while (srclen > 0)
} // end of while (restlen > 0)
if (part==0) // Single part message
{
fprintf(fDes,"=yend size=%ld crc32=%08lx \r\n",filelen,crc_val ^ 0xFFFFFFFF);
}
else // Multipart message
{
// This does not yet support the full crc32 for the entire file
fprintf(fDes,"=yend size=%ld part=%d pcrc32=%08lx \r\n",filelen,part,crc_val ^ 0xFFFFFFFF);
}
if (part==lastpart) print("Last part encoded\r\n"); // Only dummy
return(0);
}
int crc_tab[256] =
{
0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f, 0xe963a535, 0x9e6495a3,
0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988, 0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91,
0x1db71064, 0x6ab020f2, 0xf3b97148, 0x84be41de, 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7,
0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec, 0x14015c4f, 0x63066cd9, 0xfa0f3d63, 0x8d080df5,
0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172, 0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b,
0x35b5a8fa, 0x42b2986c, 0xdbbbc9d6, 0xacbcf940, 0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59,
0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423, 0xcfba9599, 0xb8bda50f,
0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924, 0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d,
0x76dc4190, 0x01db7106, 0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433,
0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d, 0x91646c97, 0xe6635c01,
0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e, 0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457,
0x65b0d9c6, 0x12b7e950, 0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65,
0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 0x4adfa541, 0x3dd895d7, 0xa4d1c46d, 0xd3d6f4fb,
0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0, 0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9,
0x5005713c, 0x270241aa, 0xbe0b1010, 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f,
0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17, 0x2eb40d81, 0xb7bd5c3b, 0xc0ba6cad,
0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a, 0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683,
0xe3630b12, 0x94643b84, 0x0d6d6a3e, 0x7a6a5aa8, 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1,
0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb, 0x196c3671, 0x6e6b06e7,
0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc, 0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5,
0xd6d6a3e8, 0xa1d1937e, 0x38d8c2c4, 0x4fdff252, 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b,
0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55, 0x316e8eef, 0x4669be79,
0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236, 0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f,
0xc5ba3bbe, 0xb2bd0b28, 0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d,
0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 0x9c0906a9, 0xeb0e363f, 0x72076785, 0x05005713,
0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38, 0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21,
0x86d3d2d4, 0xf1d4e242, 0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777,
0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 0x8f659eff, 0xf862ae69, 0x616bffd3, 0x166ccf45,
0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2, 0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db,
0xaed16a4a, 0xd9d65adc, 0x40df0b66, 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9,
0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605, 0xcdd70693, 0x54de5729, 0x23d967bf,
0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94, 0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d
}
;
int crc_val;
long crc_anz;
void CrcInit()
{
crc_val = -1L ;
crc_anz=0L;
}
void CrcAdd(int c)
{
unsigned long ch1,ch2,cc;
/* X^32+X^26+X^23+X^22+X^16+X^12+X^11+X^10+X^8+X^7+X^5+X^4+X^2+X^1+X^0 */
/* for (i = 0; i < size; i++) */
/* crccode = crc32Tab[(int) ((crccode) ^ (buf[i])) & 0xff] ^ */
/* (((crccode) >> 8) & 0x00FFFFFFL); */
/* return(crccode); */
cc= (c) & 0x000000ffL;
ch1=(crc_val ^ cc) & 0xffL;
ch1=crc_tab[ch1];
ch2=(crc_val>>8L) & 0xffffffL; // Correct version
crc_val=ch1 ^ ch2;
crc_anz++;
}
Example Decoder (in C)
// Created by Juergen Helbing in 2001 as a part of the MyNews project
// This is a source code example for the yDecoder - see also: http://www.yenc.org
// It is public domain - as well as the yEnc specification
//
// The progamming style is very "basic" and readable.
//
// The function is scanning the incoming file stream (fIn) for yEncoded blocks.
// The stream must be supported by the newsreader - in case of multipart
// the entire parts must be in the input stream. The newsreader should provide
// the decoder with a textfile which contains all the parts. This source
// can contain additional headers between the parts - they are simply skipped.
// This example requires the parts in correct sequence (1-2-3-4-...-n)
// Advanced decoders might be able to operate on mixed sequence - and even
// missing parts or parts from different "fill" posts
// print() and eprint() are logging displays from 'MyNews'
// Crc32 - functions are available in the yEncoder example
// ad_fgetscr() read data from the input stream into the buffer and removes CRLF.
unsigned long hex_to_ulong(char * text) // Because strtol() does not deliver 32 bit on my C-Compiler
{
unsigned long res;
unsigned char c;
if (text==NULL) return(-1);
res=0;
loop:
c=*text; text++;
if ((c>='0')&(c<='9'))
{
res=(res<<4)+((long)(c-48) & 0x0F);
goto loop;
}
if ((c>='A')&(c<='F'))
{
res=(res<<4)+((long)(c-55) & 0x0F);
goto loop;
}
if ((c>='a')&(c<='f'))
{
res=(res<<4)+((long)(c-87) & 0x0F);
goto loop;
}
return(res);
}
int yDecode(FILE * fOut, FILE * fIn, long y_line, long y_size,int y_part)
{
unsigned char srcbuf[4100];
unsigned char desbuf[4100];
unsigned char * srcp;
unsigned char * desp;
int deslen;
int srclen;
unsigned char c;
int id;
char * cp;
long decolen;
long y_begin;
long y_end;
long sumlen;
int partnr;
unsigned long crc32;
char name[260];
int esize;
if (aDsp) print("yDecoder started...\r\n");
sumlen=0; partnr=1;
part_start:
CrcInit(); // Analyse only CRC per part
decolen=0;
deslen=0; desp=desbuf;
if (y_part) // This is a multipart message !
{
cp=ad_fgetscr((char*) srcbuf,4097,fIn); // fgets especially with ad_length
if (cp==NULL)
{
eprint("Unexpected eof in yEncoded file\r\n");
return(1);
}
if (aDsp) print("=ypart-line: %s\r\n",srcbuf);
if (strncmp((char*) srcbuf,"=ypart ",7))
{
eprint("Missing =ypart line in yEncoded multipart message\r\n");
return(2);
}
cp=strstr((char*)srcbuf,"end=");
if (cp==NULL)
{
eprint("Missing end= in yEncoded multipart message\r\n");
return(2);
}
y_end=atol(cp+4);
cp=strstr((char*)srcbuf,"begin=");
if (cp==NULL)
{
eprint("Missing begin= in yEncoded multipart message\r\n");
return(2);
}
y_begin=atol(cp+6);
if (aDsp) print("part-begin: %ld\r\n",y_begin);
if (aDsp) print("part-end : %ld\r\n",y_end);
}
loop:
cp=ad_fgetscr((char*) srcbuf,4097,fIn); // fgets especially with ad_length
if (cp==NULL)
{
eprint("Unexpected eof in yEncoded file\r\n");
return(1);
}
if (strncmp((char*) srcbuf,"=yend ",6)==0)
{
if (aDsp) print("Endline (%d bytes): %s\r\n",decolen,srcbuf);
goto end_of_file;
}
srclen=strlen((char*)srcbuf);
if (srclen<y_line)
{
if (aDsp) print("Last line.\r\n");
}
srcp=srcbuf;
loop2:
c=*srcp; srcp++;
if (c==0)
{
goto loop; // End of line reached
}
if (c == '=') // The escape character comes in
{
c=*srcp; srcp++;
if (c==0) return(2); // Last char cannot be escape char !
c=(unsigned char)(c-64);
}
c=(unsigned char)(c-42); // Subtract the secret number
*desp=c; desp++; deslen++; decolen++;
CrcAdd(c);
if (deslen>=4096)
{
id=fwrite(desbuf,deslen,1,fOut);
if (id != 1)
{
eprint("Error in writing decoded file (code=%d)\r\n",errno);
return(3);
}
deslen=0; desp=desbuf;
}
goto loop2;
end_of_file:
if (deslen>0) // Empty the last buffer
{
id=fwrite(desbuf,deslen,1,fOut);
if (id != 1)
{
eprint("Error in writing decoded file (code=%d)\r\n",errno);
return(4);
}
}
cp=strstr((char*) srcbuf,"size="); // Compare the decoded size to the =yend size
if (cp)
{
esize=atoi(cp+5);
if (esize != decolen)
{
sprintf(name,"%s(size=%ld)",attname,decolen);
strcpy(attname,name);
eprint("Corrupt yEnc binary - endsize mismatch (%s%s)\r\n",attname,attext);
return(0);
}
}
// Check the srcbuf for the CRC
if (y_part==0)
{
cp=strstr((char*)srcbuf,"crc32=");
if (cp)
{
crc32=hex_to_ulong((char*)(cp+6));
ad_rescrc=crc_val ^ 0xFFFFFFFFl;
if (aDsp) print("Included CRC: $%08lx - calculated CRC: $%08lx\r\n",crc32,ad_rescrc);
}
}
else
{
cp=strstr((char*)srcbuf,"pcrc32=");
if (cp)
{
crc32=hex_to_ulong((char*)(cp+7));
ad_rescrc=crc_val ^ 0xFFFFFFFFl;
if (aDsp) print("Included CRC: $%08lx - calculated CRC: $%08lx\r\n",crc32,ad_rescrc);
}
}
if (cp!=NULL)
{
if (crc32 != ad_rescrc)
{
sprintf(name,"%s(crc=$%08lx)",attname,ad_rescrc);
strcpy(attname,name);
eprint("Corrupt yEnc binary - CRC mismatch (%s%s)\r\n",attname,attext);
return(0);
}
}
if (y_part==0) // Single message
{
if ((y_part==0) & (decolen != y_size))
{
// eprint("Y-Decoder: Size mismatch - file corrupt.\r\n");
sprintf(name,"%s(len=%ld)",attname,decolen);
strcpy(attname,name);
eprint("Corrupt yEnc binary - size mismatch (%s%s)\r\n",attname,attext);
return(0);
}
ad_reslen=decolen;
ad_rescrc=crc_val ^ 0xFFFFFFFFl;
if (aDsp) print("yDecoder: Job done. %ld bytes written. CRC: $%08lx \r\n",decolen,ad_rescrc);
return(0);
}
// Multipart message
if ((y_part>0) & (decolen != (y_end-y_begin+1)))
{
eprint("yDecoder: Part size mismatch - file corrupt.\r\n");
return(6);
}
sumlen=sumlen+decolen;
if (sumlen == y_size) // Completely decoded
{
if (aDsp) print("yDecoder: Multpart Job done. %ld bytes written\r\n",sumlen);
ad_reslen=sumlen;
ad_rescrc=crc_val ^ 0xFFFFFFFFl;
return(0);
}
if (sumlen>y_size)
{
eprint("yDecoder: More data than expected. File corrupt.\r\n");
return(7);
}
// Now scan the file for the rest !
partnr++; // Now scan for this part number
ploop:
cp=ad_fgetscr((char*) srcbuf,4097,fIn); // fgets especially with ad_length
if (cp==NULL)
{
eprint("Unexpected eof in multipart yEncoded file\r\n");
return(1);
}
if (strncmp((char*) srcbuf,"=ybegin ",8))
{
if (aDsp) print("yEnc skipped: %s\r\n",srcbuf);
goto ploop;
}
cp=strstr((char*)srcbuf," part=");
if (cp==NULL)
{
eprint("No part in next ybegin found - in multipart yEncoded file\r\n");
return(1);
}
if (atoi(cp+6) != partnr)
{
eprint("Unexpected part (%d) found in yEncoded message. Expected was (%d)\r\n",atoi(cp+6),partnr);
return(1);
}
if (aDsp) print("Found part # %d\r\n",partnr);
// Additional plausi possible for name & size
goto part_start;
}
// ...... A part from the decoder which scans messages for attachments .......
is_yEncoded:
if (strncmp(line,"=ybegin ",8)==0)
{
cp=strstr(line,"name=");
if (cp==NULL) goto loop; // Error - filename not found
strcpy(attname,cp+5); // Store the filename
*cp=0; // throw away the filename
cp=strstr(line,"size=");
if (cp==NULL) goto loop; // Error - size not found
y_size=atol(cp+5);
cp=strstr(line,"line=");
if (cp==NULL) goto loop; // Error - linelength not found
y_line=atol(cp+5);
y_part=0;
cp=strstr(line,"part="); // Check if this is a multipart message
if (cp)
{
y_part=atol(cp+5);
if (y_part != 1) goto loop; // MUST start with the first part for proper decoding
}
// Now we can decode the file -- let's assume that this is a valid file
attext[0]=0;
cp=strrchr(attname,'.');
if (cp)
{
strcpy(attext,cp); *cp=0;
}
sprintf(filename,"%s\\decode.tmp",datadir);
if (aDsp) print("DesFileName = (%s%s)\r\n",attname,attext);
// 27-03-2001: Set the filename jsut in case we cannot decode it
sprintf(ad_desname,"%s\\%s%s",ad_despath,attname,attext);
// filelen=0;
if (ad_crcmode) CrcInit();
if (ad_cleanmode)
{
ad_cleanmode=2; // This will prevent us from writing more to the clean buffer
if (aDsp) print("Begin cleaning...\r\n");
CrcInit();
}
fOut=copen(filename,"wb");
if (fOut==NULL) // Possibly already exists
{
eprint("AutoDecode: Cannot create DesFile (%s)\r\n",filename);
cclose(fIn);
rb_picerr++;
return(-1);
// goto loop;
}
// Now decode that file !
id=yDecode(fOut,fIn,y_line,y_size,y_part);
if (id)
{
eprint("yDecode failed (reason: %d)\r\n",id);
goto abortus;
}
print("yDecode successful\r\n");
cclose(fOut);
filelen=ad_reslen;
goto file_ok;
};