Drew Scott Daniels’ Blog

So today I got a new USB key. It's a Kingston Data Traveller. I decided
that I should set it up so that I can use it with the laptop I use on a
regular basis (a very old Toshiba Satellite 310CDS that's rattling). My
first step was to consider the file system format. I was surprised to
find that I couldn't format the device in NTFS format. I left it
formated at the default FAT32 and decided to get on to other things.

At home tonight I spent some time downloading the drivers for the device
and attempted to install them. The driver installer is an installshield
created one that's been winziped into a self extracting file (sometimes
called an sfx). Three layers of compression and installer junk managed
to make the under 49KiB of driver files take over 1MB. The second
frustrating thing I ran into is the installer is designed to detect the
operating system, and refuse to install if it doesn't think it'll work.
Well, I guess before that I had read a faq from Kingston saying that
Windows 95 doesn't support USB drives at all.

Before I get ahead of myself again, I'll go back and say that I tried to
do some research on what Windows 95 supports in the way of USB drives. I
didn't manage to find much, but I did find the usual indications that
earlier versions of Win95 didn't have any USB support, or that it wasn't
working. I already knew that USB support worked on my computer.

I guess I should have seen things coming ahead of time. In December I
had looked at trying to get pictures off a Fijitsu FinePix digital
camera. It too had an annoying installer, and claimed not to work with
Win 95. It further had a bunch of software bundled with it's
installation that I still haven't bothered to figure out. Luckily, the
installer for the driver itself wasn't hard to find, and I managed to
get the device driver, and some of the software installed.

Despite getting things installed for the FinPix camera, the software
complained about a missing dll function, and the driver didn't seem to be
working. I decided that with my many licences of Windows, I should try
to upgrade certain dll's with versions from newer versions of Microsoft
Windows. My results were of course that some of the important dll's
could not be replaced.

That got me thinking again about getting open source replacements for
certain components. I looked for a while, and decided that without a
better understanding, I might end up accidentally installing a dll that
needs an Linux shared library (.so) or something. My following of the
Wine Weekly News (WWN) on http://www.winehq.com and reading the ReactOS
developers/kernel mailing list indicated that some dll's from these
projects were defiantly dependent on components that I'm not ready to
replace.

So more recently (getting back to the USB key), I did another search on
the subject of replacing Microsoft Windows 95 dll's with OpenSource
compatible versions. I'm also now considering replacing the kernel and
other core files. I did find that WWN shows that they've been building
PE versions of their dll's for Win32, but it's not clear which can
replace the dll's in Windows 95. I get the impression that files from
ReactOS might be a better replacement than Wine's as they'll have less
Linux, BSD, Solaris related stuff in them and be created with binary
compatibility in mind for even more core pieces (e.g. no required
wineserver).

To date I've had no luck with either the USB key or digital camera under
Windows 95. I've decided that in order to start replacing Win95 on this
notebook, I'd better get a better understanding of the dependencies and
compatibilities of different components. To do this I'd like to get or
create a list of files, a graph (tree?) of the dependencies between
files, and a fresh compatibility status of the files from whatever
source I choose. Unfortunately ReactOS's compatibility page doesn't jump
out at me in searches (I remember seeing it once or twice). I also
believe both ReactOS and Wine don't list their compatibility in relation
to Windows 95, but to whatever the latest version of the component is.

So the processes I'll probably want to take will start with listing the
operating system files on the computer I'm targeting. Then I'll probably
use something like dependency walker (depends.exe from systeminternals?)
to figure out the dependencies of each files (as best I can). Then I'll
look at the compatibility status on the web. Last I may have to look at
the exports from both files. Since no one else seems to have published
this information, I'll probably write up my findings as I go. I might
even make it easier to install Open Source replacement components for
other versions of Windows by performing the same process using fresh
installs of other versions.

It's getting late now and I'm getting tired. I was planning to also
write about how to use unshield and winzip to extract files from
annoying installers. I also felt the need several times to explain why I
wanted open source replacement files, and didn't upgrade Windows
(remember I do have licences to newer versions). I guess I can quickly
say that I like having free access to the source of what I'm using so
that I or just about any other programmer can enhance/fix it. I also
don't want to install Windows98 or later on this laptop because it may
take more system resources, not run, and well I'd rather maximize the
use of my Windows 95 licences before using other ones. I've tried
ReactOS and Wine, and I know they're still not 100% replacements for
Windows (although extremely close nowadays). I also believe that other
people share my viewpoints and/or situations.

Maybe later this week I'll write more on the topic of replacing windows
components or Windows Device Drivers (wdm, ndis, inf, the wonderful
dpinst.exe and more), but for now it's time for me to get some sleep...

Originally from: http://www.boxheap.net/ddaniels/notes/20050817.txt

I was planning on writing about the problems I faced at work looking up
open source software for SMPTE 125M convertion. I kept finding SMPTE
timecode stuff (for MIDI), and other usages of the acronym SMPTE without
reference to which standard was being used. The one's related to SMPTE
125M are SMPTE 292M (HD-SDI), SMPTE 259M (transport of SDI and SDTI),
SMPTE 305M (sometimes called SMPTE 305.2M which is SDTI), and the
document on ancellary data. Actually SDTI really is quite different from
SDI except that it goes over 259M.

Anways, tonight I think I write a bit about linking and google. Yes,
part of the reason that I'm writing these notes is to increase the
ranking that I'll get for topics that I'd like employers to see. The
bigger way that I plan to get a good ranking is something I've
accidentaly found before. I've put a one line signature in my e-mails to
mailing lists with my resume's URL. I was hoping I could find someone on
the mailing list that might be interested, or might refer me to someone,
but instead I found that the html mailing list archives looked to be
increasing the rank of my resume. I guess this was a neat trick that can
work on google, and maybe on other search engine's that look at what's
linking to a page to give it a score.

When I finnaly am happy with the testing scripts that I'm working on for
my tarball enhancements I'll post the results to various mailing lists
that are development forums for projects with large tarballs (e.g. the
lkml, some kind of gimp mailing list, maybe some OpenOffice.org AKA OOo
mailing lists...). I've got my resume's URL in the scripts themselves,
but I also plan to put my resume URL tagline in my messages.

One of my problems with my tarball enhancement postings is that I'll
want a perminate place with my domain name that I can host the scripts,
but I'm getting free hosting from a friend (thanks Dean). I don't want
to generate a lot of hits on my friend's server due to the fact he
likely has better uses for his bandwith, and his ISP may not apreciate
it. To prevent such a load on the link to his server (and his server), I
plan to keep the scripts only on the mailing lists (archived in their
archives) until interest drops down a bit. I figure a few weeks would
do, but I'll probably wait a few months.

I'm really quite kean to get my scripts out the door, but I feel they're
not yet ready to stand up to the kind of critism that one gets on the
Linux Kernel Mailing List (lkml). I've got a script to do the actual
tarball creation, and one to show the difference between a normaly
generated one, and the one my script makes, but I don't have something
showing the amount of time that it takes. Measuring the sorting isn't
easy, as it's a series of piped commands. My shell scripting really
isn't put to enough use for me to be able to quickly work around such a
problem. I've checked a few howto's like the bash one, I've asked in the
bash scripting IRC channel, but I couldn't find an answer. I decided to
put the commands into a separate script and time that whole script.

The other problem I've run into is testing. My home computer was taking
a beating compressing and untaring etc.. I decided to use my
SourceForge compile farm shell to do the testing, but it's a pain to put
files onto them. It took me a while before I figured out I had to
download the files to my computer, and then upload them to the compile
farm's central server via sftp or scp. That's something I can do, but it
really compounds another problem I'm having. It takes me a while to make
progress on my free time coding projects, so new target files are
comming out for me to test. I want to be able to post on the lkml the
results of recompressing the latest 2.6 and 2.4 kernels. I keep
optimistically downloading the latest kernels and then having real life
interupt things long enough for me to need a new version to continue.
I'll stop doing that for a while though until I've actually got a draft
sitting in my posponed box of an e-mail to the lkml with the scripts
already finnished and attached or actually inline I think. That's
another problem. The lkml only accepts certain posts, and Linus only
usually accepts things that are in a certain format (plain text inline
iirc). That put me on a tangent of looking up the mailing list rules,
and reading the Linux Weekly News. It'll likely do the same once I get
close enough again.

So with all my knowledge, reading, and interest in digging deep into
open source stories that I see writen/posted, I've thought about trying
to get payed to write. These notes are a bad example of my ability to
write, but a good example of what I enjoy writing about. I've been
solicited once to write a book on Intrusion Detection from a genuine
publisher, but I kind of "fubbed" my responce. I said that I'd be
interested in contributing, but I didn't think I'd have time to write a
whole book. I kind of regret doing that, but I think it was the right
thing to say (just look at my bad record finding time to do coding). I'm
hoping however that a paying gig would actually let me take some time
away from real life to actually get things done (and I'm sure it would).
Of course I've got to stike a balance to keep my home life happy and
healthy (fammily, friends, and my own condition). I've offered to write
a peice on the history of the BSD's to the Linux Weekly News, but they
didn't seem interested. They do post BSD articles, and I was pitching
that I could write one that would show the parallels between AT&T vs The
Regents of Berkly (BSD) and the current SCO vs IBM etc.. It's
interesting how the history repeats itself. For good reference I'd
suggest reading the FreeBSD mailing list archives (a google search found
some good stuff).

Later I might publish the research that I used as part of my pitch for
my BSD history repeats itself story. I'm also probably going to consider
writng about why I don't want to publish my unrealized ideas. I'll also
probably talk about:
- Why I don't write about office politics
- Why I don't write much about my personal private home life (well,
maybe I made that clear <g>)
- My music idea's
- My thoughts and research into a self powered home (well actually
getting power form alternate sorces like sun, wind, water...)
- Thoughts on using "image stacking" for ameture (and hopefully
professional) astronomy (I'll talk about this because other people have
already implemented some of this)
- Some idea's for how people can generate data that's easier to compress
(e.g.'s typing in lower case when there's the option, removing obvious
redundant information, using the same words...)
- Perhaps my ideas on natural language processing
...

I may eventually post my project ideas from the last fourteen years that
I've been writing on paper.

Consider sending me money! My resume is at
http://www.boxheap.net/ddaniels/resume.html

Oh, and I'll probably write about resume creation and open source tools
to do it (hey, maybe lwn.net would be interested in buying that
article).

Originally from: http://www.boxheap.net/ddaniels/notes/20050811.txt

There doesn't appear to be any adopted standards for MPEG over IP. IP
over MPEG looks more interesting. Just packetize an IP stream into a
packetized elementary stream (PES) and multiplex it into a valid MPEG2
Transport Stream. MPEG-2 typically gets transfered over DVB-ASI, DVB-C,
DVB-SI, DVB-T and other protocols (even "ATSC" AKA SMPTE 310M).

So how do you packetize IP packets to go into an MPEG stream? Well that
depends on the source. I'd like to think that any IP source "worth it's
salt", is from a live network. Thus a network feed would need to input
into the packetizer, multiplex it and put it out over a different type
of device. I've heard of some people making a network device driver for
DVB-ASI cards, but at least one engineer I talked to said there's
probably a better way. He suggested keeping the regular characteristics
of the ASI device, and doing the packetizing in application space. I
managed to convince him however that the conveniences of creating a
network device which can be bridged would be far better. He stuck with
the separate device driver idea however and suggested one driver could
use the other.

So then the question is, how do you create a network device driver
that's just a packetizer, multiplexer and forwarder? No doubt there's
some good examples out there, and NDIS should make it easier. I still
worry about doing more than elementary processing in a driver might
cause some strange system behavior. I guess I should also say there's
probably an even easier way to do things in Linux and FreeBSD variants,
but I'm mostly focused on the Microsoft world as that's what I'm told by
Marketing is what's wanted.

On the opposite end you need a depacketizer? Or something to demultiplex
the stream, and put IP back out onto the network. I've seen this done in
software, and that might make more sense on this side of transfers. The
engineer that I speak of above however suggested that the unidirectional
nature of MPEG II transport streams would give another problem,
associating one direction of traffic with the other.

I'm not quite sure how other people bind one transfer direction with
another, but I remember several satellite companies offering service
that beamed high speed broadband internet access to customers and
accepted data back to them via telephone modem. So schemes to put two
different directions from seperate devices have been around for a while.
I just hope that modern network stacks are smart enough to remember that
it's allowed.

I remember someone telling me that the ARPA network was an experiment
designed with the goal that it be able to stay up, even if one link in
the network went down. It failed, or at least that's the punchline. The
modern Internet can't reroute if there's a failure in a router. There
was a fire in a telecom building in Toronto, and connections from
Manitoba Telephone Services (MTS) to Shaw in Winnipeg went down. I've
also seen where an outage in Shaw's network caused places to become
inaccessable, but if you had a proxy on a CA network accessable address,
you could access the rest of the internet. Those are just two local
examples that I know about. The CA network thing is political (I'm told
they're not allowed to carry commercial data due to their funding
grants). For what it's worth, I've also seen many shares of
misconfigured routers, the more obvious cases were with major telecom.
companies.

So back to MPEG transport streams. I know companies like Norsat have
been selling "solutions" to do these things for years, so I think
there's a market. Identifying the market potential is difficult for this
because it's not something most broadcasters, stations, and local
distributors are looking for. It's also not something that's really even
remotely accessable to consumers.

A similar issue that I've thought about for even more years is multiple
links between computers to increase throughput. I know lots of other
people have looked at bridging and bonding, but I wanted to look at it
at an even more insane level, serial ports. Actually I wanted to look at
paralell ports, modems, ethernet, etc.. I suppose it is possible to bond
all these links together, but it certainly isn't common enough that it's
as easy as listing the links (at least as far as I know).

So why bother with all this legacy stuff? Why not build a new network
card that can communicate at the full buss speed? Well actually we're
pretty close to that now. From my own experiences I've calculated that
modern HD-SDI cards must be close to maxing out the bus throughput.
I've also learned that multiple cards on the same bus can't allow a
faster network connection as of course there's only the one bus. Of
course I've seen computers with multiple bus's, but it's hard to know if
they're truly independent, or if they're more likely bridged. Even a
bridged network of bus's can allow each bus to operate almost
independently, if the parent bus is faster than it's children combined
there may be an advantage to using multiple cards.

Even further it's important to note that most modern bus's have
bottlenecks. When was the last time you looked up the DMA latency of the
motherboard you wanted to buy? I'll wager never. I've thought about how
useful this could be to consumers and whether there was a way I could
get the comany that I'm working for to publish regular results of DMA
throughput and latency. We could then get free motherboards. The idea
likely wouldn't work though as that's not really what the business
does.

Other crazy idea's I've had include using every processor on the system
to do computations including the IDE/ATA hard drives (they have RAM
too!). Alas of course most of it would be very convoluted to figure out
a way to use.

The more recent idea that I've had (since Mark Nelson's "random" binary
file challange was posted), was to figure out a list of common
instructions and library calls for which I could get more output that
would be required to issue the request for data (e.g. more bits from
register results than from instruction cost...). This idea has some
potential, but my current "hurdle" is finding the time to get and go
through a list of CPU instructions. Getting a list of available function
calls is also a challange although maybe a nice program to do it already
exists (just get the exports from all dll's etc?).

So as you might be beginning to see, one of my primary interests is
data compression. I used to be interested in the pure pattern finding,
and making the smallest representation possible of common data, but then
I started working in multimedia. It became obvious very fast that the
speed of compression actually is important (not just to those who can't
wait). If things don't compress fast enough you can get overruns, data
loss and ultimatly data corruption (even if that just means missing
bits/bytes/frames...).

One of my past pet projects was zlib compression of SDI (see StreamBed's
deflate option). I've found that I can gzip at 270,000,000 bps (that's
270Mbps in SI notation). The small b means bits of course. The problem
is that it ether needs a fast processor, or a simple pattern (like
colour bars). It may even need both. I haven't had the time to check.
Unfortunaly without the inflate option in StreamBed, customers aren't
too interested yet, and without customer interest my boss isn't too
interested yet either.

I later plan to talk about:

compression of MPEG transport streams (they're already MPEG
compressed, but the tables are text, and there's that predictable 0x47
once per packet or 188/204 bytes).
lossy MPEG table compression (recreate it to meet spec)
Alternate SDI compression
Compression in firmware
My literly magic tarball scritps to improve general tar.gz and
tar.bz2
compression with an order(1)+small sort preprocessor
My extensive prediction by partial match (PPM) research and
documentation (I've spent at least the last 5 years working on a new
algorithm that I have high hopes for, but strange results)
My ideas (hopefully implemented soon) for a very simple honeypot
like intrusion detection system
problems with current network security tools
Perhaps products and development directly related to my jobs


Drew Scott Daniels' Resume
Originally from: http://www.boxheap.net/ddaniels/notes/20050810.txt