Drew Scott Daniels' Blog Personal, usually technical posts

November 30, 2015

NEO mining

Filed under: technical,Uncategorized — admin @ 8:49 pm

From my paper notes from years ago:
* Find Near Earth Objects (NEO) catalog(s)
* Determine what is of value [e.g. precious metals]
* Make qualifications list
* Find candidate and rank by value, cost (capture and building), and both
* Pick optimal targets
* Devise methods of capture & construction
* Present plans to backer(s)

I recall thinking that near earth asteroids would take less fuel for transport of, to and from. I also recall thinking and that asteroids have low gravity which makes it easier to escape from.

I was holding back on publishing this post, but now seems appropriate given the recent passing of the US “2015 Space Act”. Details are at: https://en.wikipedia.org/wiki/Asteroid_mining#Legal_regimes

Other recent news includes NASA’s Near Earth Asteroid Scout (NEAScout http://www.jpl.nasa.gov/cubesat/missions/neascout.php ) which is going to launch as a secondary payload on the 2018 Space Launch System (SLS) Exploration Mission 1 (EM-1) (story at: http://www.nasaspaceflight.com/2015/11/nasa-identifies-secondary-payloads-sls-em-1/ ). This mission should help figure out what other Near Earth Asteroids are made of.

Drew Daniels
Resume: http://www.boxheap.net/ddaniels/resume.html

November 22, 2015

Minimal backups of a Debian or Ubuntu system

Filed under: Data Compression,technical,Uncategorized — admin @ 11:36 pm

Storage has become so cheap for me that I now use bacula and do full backups based on a broad file set definition. For offsite backups I focus on a smaller set of more important data. At work there are different kinds of requirements and restrictions too.

Back in 2002 I was thinking about minimal backups and I was dealing with quite a bit more software not following the Filesystem Hierarchy Standard A post about my thoughts is at https://lists.debian.org/debian-devel/2002/07/msg02232.html with Message-id <Pine.GSO.4.40.0207311107350.16701-100000@castor.cc.umanitoba.ca>. The same kind of methods can be used on Ubuntu too.

Example minimal backup scripts for Debian:
http://www.linux-backup.net/scripts/aaron.sh.txt
http://qref.sourceforge.net/Debian/reference/examples/backup

I wouldn’t ignore files that I installed that are in special directories and are not packages. The package called cruft or perl program Debarnacle (which can be installed using cpan) can be used to find files that are not in packages. The above scripts may not be smart enough to backup everything. “cruft” has some problems when being used to figure out what files to backup, this is because it is designed to find files that can be removed. Removing unnecessary files and packages is a good first step before a backup, but don’t delete anything you’re unsure about.

I forget all my difficulties in using “cruft” to assist in creating a backup, but I’ll try to recall. One problem was a feature that it ignored directories that it knew would have files that weren’t part of packages like /home. An inconvenience was that I had to sort through all the files that it said should be there, but weren’t (perhaps files to delete after a reinstall of packages, although some I should have had). Another problem was that I had to force it to skip some of the other file systems I had mounted like CD’s and my dos partitions. “cruft” seems to keep a large backup cache of it’s database, this was sometimes helpful, sometimes I needed to delete it (maybe I did to save disk space as I remember it being big).

Debian backup instructions, and backup information (obsolete and deprecated since 2006):
http://www.debian.org/doc/manuals/system-administrator/ch-sysadmin-backup.html
Replaced with:
http://www.debian.org/doc/manuals/debian-reference/ch10.en.html#_backup_and_recovery

“dpkg –get-selections>list_of_selections.txt” can backup a Debian user’s list of currently selected packages. “dpkg –set-selections<list_of_selections.txt” to start a system restore, iff packages are available. Some packages get removed from the Debian archive (http://ftp-master.debian.org/removals.txt has recent examples), if following unstable or testing, then you may want to use dpkg-repack or grab a copy of the packages you are worried about. http://archive.debian.org may also be helpful.

A simple rsync, ssh, cron solution can do regular incremental backups, but might ignore files that are easily available (like packages on cd or packages available by a quick download).

Discussion on backing up a Debian system:
http://wayback.archive.org/web/20020821185115/http://www.debianplanet.org/node.php?id=586

My Debian backup steps:
1. I remove files and packages that I’m sure I don’t want or need (deborphan can help me figure this out. I later learned to use aptitude and apt-cache to help find reverse dependancies)

2. I run “cruft” first on all directories, trying to avoid special devices or removing checks on directories where it seems to stall (not the best way, but the best way available right now). It would also be nice to use the md5 signatures for files to see if I manually changed a file in the file system, I’d of course want to back those up (watching for hacks of course).

3. I remove files that I’m sure I don’t want or need that are indicated to me by cruft. (possibly also removing them from cruft’s report file or files to save from doing step 7)

4. I fix the list of missing files indicated from cruft’s report(s). I also possibly install packages that have files installed, but the package isn’t listed as installed for some reason (not likely to be able to skip step 7 if I do this second part).

5. I look for packages that are not available for download or available in a reliable location (Usually all obsolete packages listed in dselect, sometimes more, some obsolete packages have -src packages that they can be
built from). I then backup these packages using dpkg-repack, or if available, I grab a copy of the proper package (dpkg-repack doesn’t create packages back to the way they were originally. http://archive.debian.org may be useful to find packages that are not in the archive anymore.).

6. I run “dpkg –get-selections>/root/myselections.txt” (this file is important to backup unless you want to go through the list of packages to install again, step 7 should catch this, or you can add it manually if you skip step 7).

7. I re-run “cruft” the same way I ran it for step 2.

8. Go through cruft’s report and remove any information that I don’t want backup (maybe keep a copy of the missing files separate).

9. Create a list of files to backup using cruft’s report as a good guide (remembering to include myselections.txt and any important packages). /etc and all the conf file information under /var/lib/dpkg/info may be good to have (not including unnecessary files would be nice).

The rest of the steps do not allow for incremental backups, and may be modified to allow incremental backups.

10. Append “tar -af backupfile.tar ” in a text file, at the beginning of every line that lists a file to backup except the first one which I do “tar -cf backupfile.tar” (tr may be helpful, but what’s the proper command? I’d prefer to avoid perl, but is it more common than tr? If so what’s the proper perl -e line?). I then make the text file executable and execute it.

11. I ran “bzip2 -9 backupfile.tar”. (p7zip might be one of the best choices for the time of this post)

12. I used xcdroast or cdrecord or something else to burn backupfile.tar to a recoradable cd. Brasero is more commonly used these days and burning to a DVD though USB and hard drives are getting more common. I currently use a porable USB hard drive and network attached storage.

I appreciate the help I’ve gotten so far in generating a program/script to automate these steps (and getting it made into an uploaded package). Help in figuring out a good way to make an incremental backup would be very useful. I think there must be a nice way to do it with tar and freshen, or using the archive bit.

Some packages get removed and the latest version isn’t publicly archived. I don’t like that unmaintained or any potentially installed packages are removed from the Debian archives. I brought this up with the qa group and was refered to http://snapshot.debian.org and “the morgue” which was on auric but now seems to be at merkel.debian.org:/org/ftp.debian.org/morgue/

I worry that not all packages will have md5 signatures of their files. I remember having a problem with many files not having md5 signatures, or even some packages having incorrect md5 signatures. I think *every* file that comes in a .deb should have an md5 signature that gets installed when the corresponding package gets installed. I wanted this to be a policy must as it’s good for backups, and detecting system corruptions (hacks and modifications that are intentional or malicious or accidental). md5 signatures should take little effort to create and maintain with an archive. This would help with tripwire or aide like setups (usually designed for file system based host intrusion detecion systems also called host IDS’s or HIDS).

Drew Daniels

Resume: http://www.boxheap.net/ddaniels/resume.html

March 4, 2013

Tea pots

Filed under: Uncategorized — admin @ 7:49 pm

Glass provides a nice view of the tea inside. It’s also essential to the presentation of wrapped or folded tea such as tea flowers.

Some tea pots are designed so they can be used to boil water on a heating element (such as a stove), and can also be kept warn on a heating element. Glass and some ceramics will break if put on a heating element. A common characteristic that I’ve noticed is a metal collar that makes the tea pot look like a coffee pot, though the shape is different.

A “Brown Betty” tea pot is quite commonly used in many households, though with tea culture from various sources spreading, other pots are often seen too. One appealing aspect of the common colour used for the tea pot is that it doesn’t show tea staining very well.

Some designer teapots can be hand painted, have unusual shapes (such as squares and triangles), and unusual colours.
Paderno makes some nice shiny metal double walled metal tea pots which may help to retain heat.

When looking for a tea pot, consider how it will be cleaned. Built in strainers are nice, but may be difficult to clean. Also, you may want to consider the size of the top opening verses the size of your hand so that you can easily reach inside for washing the pot.

A coffee press (or even coffee pot) can double as a tea pot. I don’t recommend using it for both as coffee coating can cause a scummy residue to form around the edges of the container and float in the tea.

A small individual serving tea pot can be a nice option, especially when re-using tea leaves as is customary with some tea cultures. In such cultures, I’m told that attention is payed  to not “scalding” the tea by using water that’s too hot and using a first pot quickly drained called “washing water” which is sometimes drained into the cup to wash it out. “Washing water” also serves to “steep the pot” providing removal of some residues and warming the pot to better retain the heat of future cups. I suspect some tea leaves may need to be rinsed off too. Using this method, I recommend varying the brew/steep time according to the number of uses the leaves have seen.

Yerba Mate is sometimes consumed using a gourd and a metal filter straw. If using a gourd, it’s difficult to see the leaves. The Columbian that introduced me to using a gourd said that it’s common to re-use the leaves in the gourd. I’ve seen restaurants like La Fiesta using a coffee press to give a nice visual display of Yerba Mate. Of course with leaving leaves in, I recommend consuming the tea within the first few minutes to prevent too much caffeine from leaching out.

A good China/porcelain tea pot can provide an appealing design with matching cups, saucers, creamer, sugar bowl, serving tray etc.. The rims are sometimes decorated with a thin line of gold. While very visually appealing on their own, they can be impractical to wash due to how delicate they are, and likely should be washed before tea stains set in. I have not seen any tea stains set in China or porcelain, but at the cost it doesn’t seem worth the risk. I have seen tea stains in ceramic tea cups, even those with a glossy finish.

In one episode of Sherlock called The Blind Banker tea pots from China in a museum must be washed with tea regularly to keep from cracking, and to help them slowly get a nice tea colour. I have not heard of this outside of this tv episode and expect that such a pot would be very impractical even for an avid tea drinker.

January 6, 2012

Camp Fire Wood

Filed under: Uncategorized — admin @ 1:49 am

Some Manitoba provincial parks like William Lake have free firewood (in piles outside), and some like Birds Hill have firewood for sale ($8 this summer for a bundle of dry, covered wood). Some parks like Stephenfield have firewood for sale just outside the park.

This last summer I stayed at Pickerel Point Campground which is on Madge Lake in the Saskatchewan provincial park called Duck Mountain. The firewood was free, but was very green and was piled in heaps exposed to mud on the bottom and rain.

So what makes good firewood?

  • Clean of dirt and mud.
  • No or very little bark.
  • Old cuts so that the sap dries out. Preferably stored dry for a year.
  • Dry rotten wood burns quickly.
  • Small pieces.
  • No leaves, grass, or twigs.

Using a hatchet, you can remove thick bark and make cuts in some dryer wood. Using an axe you can cut up wood that is more wet, or larger. Other options include mechanical splitting devices (hand, pneumatic, electric or gas powered), a wedge and mallet/hammer…

Remember more air gaps in upper surface area (e.g. small pieces) means more oxygen for combustion, and more upper surface area means less ash and more fuel available to burn. This also means arrangement of small pieces is important. It also means larger, pieces burn slower which can be an advantage for longer fires.

NASA recently studied fire in space aboard the International Space Station (ISS) and since there’s no upper side to fuel, ended up with interesting results. I suspect liquid fuel benefits from dispersion of burned fuel if the burned fuel is less dense. No upper side also makes it harder to extinguish flames as fuel needs to be covered from all sides in an even form so that it doesn’t cause propulsion (think rockets, and newton’s laws of motion). This is however getting a bit off topic.
More information on NASA’s fire studies at: http://www.space.com/13766-international-space-station-flex-fire-research.html and http://www.nasa.gov/mission_pages/station/research/experiments/FLEX-2.html and other places.

Drew Daniels

September 18, 2011

Tea and related infusions

Filed under: Uncategorized — admin @ 4:33 pm

There is only one kind of tea plant, but many infusions. Black, green and white tea are just oxidized for different amounts of time. Other interesting related infusions include:
* “Red tea” (Rooibos) which is made from the roots of rooibos bushes.
* Yerba Mate
* Mint “tea” which is just mint or spearmint infused in water
* Lemon “tea” which is commonly made with lemon grass

Common brewing/steeping instructions for tea:
1 Clean the pot.
2 Use good dechlorinated, preferably filtered water
3 After the kettle has boiled, pour some water into the tea pot, swish it around, and let it sit for a short time. I sometimes time it by adding more water to the kettle, and boiling the kettle again.
4 Pour out the water out of the tea pot. You now have a “steeped” pot.
5 Use one bag/tea spoon for the pot and one per person. With most tea one bag/tea spoon is roughly two grams. Obviously there’s more factors.
6 Wait two to three minutes, and pour into tea cups.

Chinese preparation of tea includes “washing water” to wash the tea leaves, and doesn’t leave the tea sitting in the pot for long. The more common practice in that case is to use small one cup tea pots, and add more water before making a new cup. Common tea used for the style is large leaf (i.e. not cut up or ground) green teas, though white tea is highly regarded.

There is a British standard for brewing tea. It is “BS 6008:1980”, now ISO 3103. This standard is similar to instructions on many tea’s, but brew time six minutes.

Tea has less caffeine than coffee, but that’s measured for standard brew times. The long the brew/steep time, the more caffeine released. I suspect how ground the tea is effects how much flavour and maybe caffeine is released into the water.

Red Rose tea has 3 grams of tea per bag versus the more common 2 grams. Red Rose tea also seems to use tea leaves that have been ground up more

Ceylon tea comes from what was once the country of Ceylon and is now called Sri Lanka. Assam tea comes from the Assam province of India. Tea comes from Assam trees, and other varieties of the same species.

In India, tea is commonly known as Chai. Translating Chai tea into English makes it seem as funny as it is: Tea tea. Traditional recipes for tea in India, and other places are more accurately called spiced tea. Extending the Table has a good recipe for spiced tea. Several people I know say that Red Rose tea is the only tea to use for making good spiced tea, and that it’s best made on a stove with milk instead of water.

Real Earl Grey tea is more than black tea with bergamot flavour/oil. One commonly missed ingredients is lapsang souchong which is black tea that has been smoked with cedar.

A good London Fog is made with real Earl Grey, brewed in a pot of milk, with good quality vanilla extract. Real vanilla tastes noticeably different than artificial, but is twice the cost. Watch out for real vanilla extract that has a low quantity of vanilla in it. I suspect the extracting agent of vodka/alcohol and the additive of sugar can make a significant difference in vanilla extract.

Tea tastes best when it’s fresher. To keep the flavour of tea, keep it in sealed containers, away from heat and moisture.

I like pre-bagged tea the best at home and work as it’s easy to clean up. “Coffee” presses, various tea balls, tea strainers can be used, but removing finer ground leaves can involve more work than cleaning regular dishes.

Yerba Mate is a nice tea like infusion. It’s great by itself, but is also nice with honey. Make sure to use good honey though as some store bought brands of honey taste noticeably worse than good local clover honey.

Some of my favourite teas and infusions include:
* Earl Grey
* Yerba Mate
* London Fog
* Darjeeling
* Common black teas (Red Rose, Assam, Ceylon…)
* Prince of Wales
* Black Currant
* Lapsang Souchong
* Sweetened/unsweetened lemon iced tea.

Quality ingredients that can be added:
* Manitoba clover honey (it’s local to me)
* Good quality vanilla extract (I’m having trouble finding a good source)
* Whole spices to grind when making spiced tea
* Raw sugar
* White sugar (it’s the sweetest additive)
* Fresh milk (I like 2%)
* Lemon juice (don’t mix it with milk. I use lemon extract for convenience.)

Drew Daniels

August 21, 2011

Self-Watering Plants

Filed under: Uncategorized — admin @ 11:20 am

Research into creating materials to allow plants to release and absorb water from stores only when needed is continuing. This research could be valuable to not only reducing water shortages, but also increasing productivity, making space exploration easier, increasing plant health (including potential yields) and helping out with long haul cargo trips. The addition of nutrients using the same method makes this even more useful.

According to a letter by Hymie Gesser of Winnipeg in an article in the Winnipeg Free Press on August 11th, 2011 work is continuing on a self-watering process for plants that uses a “special micro-porous plastic material that separated the plant’s roots from the supply of water and nutrients.”. The original process was for potted plants written about by Louis Errede, of 3M in St. Paul, Minnesota, and Gesser has extended the process to row plants and just patented it. Apparently there’s a more detailed article about Gesser’s work in the Winnipeg Free Press on May 17th, 2002. I found another article that seems to be written by the same researcher with the name variant Dr. Hyman D. Gesser. The detailed article by Gesser is available from Winnipeg Jewish Review at: http://www.winnipegjewishreview.com/article_detail.cfm?id=145&sec=2 and seems to have been written in or around 2009 as it refers to an article published in the Journal of Applied Irrigation Science, Volume 44, (2009) pp 31 -37.

Self-watering plants seems like a useful concept to reduce water usage, but it is worrying that the work was verified in Florida which I generally associate with being wet from rain and swamps. It’s also quite a hot climate. I’m also discouraged that drip irrigation seems to be preferred by several desert institutions.

Reduce water usage could be very useful to grow food and raw materials in deserts, other dry lands, or even on space expeditions. Long haul trips like cargo ships and distant space travel could benefit not only in reduce water consumption, but also potential longer life for some perishables, and less human intervention being required. One practical application for me might be for some office plants that I’m considering buying, but will not be able to water on long weekends, or some vacations. I can imagine some offices that have shutdowns for weeks, and some that have very few workers that could benefit. Also, reduce time spent watering plants means more time available for other tasks. People likely won’t deliver water as efficiently as the plants want, but hopefully a technology like this will which may mean better yields and healthier plants.

I haven’t found out what the source of the water is, but for potted plants it seems like manual refills will be required. For row plants hopefully there’s tubing that allows water to flow, though distances and rates of flow may mean that there are more opportunities for research. Fortunately Gesser lives in the same city as me so perhaps I’ll have the opportunity to find out more.

Further Research

Some topics that could be further researched include:

  • What are the substances released by plant roots when water and nutrients are needed, or not needed?
  • How can the costs of materials and production be reduced?
  • What plants are best suited to these watering techniques? Guesser found it to be effective for many different fruit and vegetable plants.
  • Do these techniques not work with certain plants?
  • How does this technique effect yield? It may reduce yield in certain cases.
  • Should nutrient or water be decreased during certain events such as certain growing phases, different weather, different ground conditions, and pesticide application?
  • How can materials be protected during tilling?
  • How do pesticides effect the materials?
  • How do “weeds” behave in the presence of the materials? Last year’s crop can be considered a weed.
  • How do insects and animals interact with the materials? This may make a good home for certain species, some may puncture materials making them less effective, and some might find the materials toxic (though this seems less likely).
  • How do various harvesting techniques effect the materials? Using a mower cuts lower than a swather. Flooding fields to harvest cranberries could be significant to the materials…
  • How can materials be repaired? Delivery of “healing” material may be possible via the same tubes.
  • How do different shapes and orientations make a differences? Tubes with branches, capsules, spikes…
  • Can roots be attached to more directly?

Further reading

Louis A. Errede and Patricia D. Martinuccl, Flow Rate of Water through Porous Membranes as Affected by Surface Modification on the Low-Pressure Side of the Membrane, 1980, 19 (4), pp 573–580 found at: http://pubs.acs.org/doi/abs/10.1021/i360076a018

João G. Crespo, Karl W. Böddeker, North Atlantic Treaty Organization. Scientific Affairs Division, Membrane processes in separation and purification, Springer, 1994, ISBN 0792329295, 9780792329299

C. A. Heath and G. Belfort, Synthetic membranes in biotechnology: Realities and possibilities, Springer Berlin / Heidelberg, 1992, ISBN 978-3-540-55551-3. Abstract found at: http://www.springerlink.com/content/245u31675230m064/

June 13, 2011

Camping supplies

Filed under: Uncategorized — admin @ 11:39 pm

Shampoo/Soap

Toiletries like shampoo can be a pain to take when travelling or camping because it can pop open, or leak needing a plastic bag and leaving half open containers. Shampoo’s number one ingredient is usually water (sometimes they call it aqua). Some bar soap from Rocky Mountain Soap is also shampoo, and I’ve heard of other soaps that can be used for shaving, bubble baths, and more. The “shampoo bar” is listed as only for men and for dry scalp, but for camping I imagine the applications might be wider, and if you visit, they might also know of other useful products. Their soap is “natural” so that may appeal to a camping crowd too. With limits on liquids for air travel having bar soap allows another toiletry to be carried on reducing what you might need to check-in if you want to make sure you have it. Of course with air travel, another popular option is getting free small bottles from the hotel or buying it along with other supplies from a store at the destination.

Food

There used to be a product that I believe was called Pasta Magic which was widely available. Pasta Magic was great for hiking or long duration camping because the pasta needed minimal preparation. Now I look to Mountain Equipment Co-Op (also known as MEC) for their camping food. There’s even some good substitutes for some of the frozen food people bring to the office. Camping food can be a great alternative where no microwaves are available. Some foods are ready to serve, and many simply require hot water. Some foods cheaper than a cheap meal out while also taking less time to get, possibly being more healthy, and many “camping” foods just aren’t available elsewhere. Unfortunately cooking directions seem to be missing from at least some of the online entries so you can’t be sure if they expect a pot and fire/stove to cook.

Note

I’m looking at helping out more with Beaver Scouts (“Beavers”) next year so there may be more camping related posts coming up.

May 24, 2011

Adding old paper notes

Filed under: Uncategorized — admin @ 11:38 pm

I plan to start adding in my old hand written notes into this blog. I have notes going back more than 13 years. Most of my notes were written during my time at the University of Manitoba.

I have notes on Data Compression, games, short-wave radio logs, computer errors, my own literary writing, course related reference notes, notes about various organizations I was in, travel observations, business process observations and more.

I’ve got a small folder of things mostly from 2004 that I may be starting with. I’m still debating what’s worthy for entry. Some notes I may have lost the context for. I must admit that one of the reasons for this effort is reducing the pile of paper I’ve accumulated.

At one point I was taking notes on my Palm device, but after several minor data loss incidents, and the nuisance of changing batteries I stopped using it. I have some notes there that I may look at putting in this blog too, but since I’ve got a digital copy there’s less urgency.

Another challenge I have is what to do with the various diagrams and pictures I’ve drawn. Most of it isn’t relevant, and when it is I may put it into a proper web page and re-draw it digitally. I think when I was younger I had a notion that my hand written notes would be treasured like those of the scientists of history, but that was long before the topic of “horders” hit the main stream media. Now my rules for what to keep and what to dispose of is leaning towards getting rid of more.

I have a small file folder filled with small notes on books I’ve looked up that I’d like to revisit and maybe post reviews about. Without looking I know that I want to re-read some books that quote “How To Solve It”. I particularly liked the following problem:
If a bear walks south 1km, east 1km, and then north 1km and ends up where it starts, what colour is it?

Drew Daniels

March 20, 2010

Random word’s definition shell script

Filed under: Uncategorized — admin @ 9:16 pm

dict "$(head -n $(($(od -N 2 /dev/urandom|cut -d' ' -f2 -s)%98326)) /usr/share/dict/british-english| tail -n1)" |less

More people may prefer:

dict "$(head -n $(($(od -N 2 /dev/urandom|cut -d' ' -f2 -s)%$(wc -l /usr/share/dict/words|cut -d' ' -f1))) /usr/share/dict/words| tail -n1)"|more

  • I still want to check for bashisms
  • 98326 is the output of wc -l /usr/share/dict/british-english. I put it inline for speed, and didn’t bother with a variable since I wanted a one line script.
  • If the dictionary is too big then a larger random number would be needed.
  • /usr/share/dict/british-english isn’t installed on many systems, but words is.
  • “more” is lighter weight than “less”, and is installed on more systems. It lacks the ability to go backwards.

To get a random number I used:

$(od -N 2 /dev/urandom|cut -d' ' -f2 -s)

  • od converts to decimal.
  • -N 2 gets two bytes
  • /dev/urandom is pseudorandom bytes from the kernel. There might be a more cross platform alternative like maybe $RANDOM for bash.
  • cut -d' ' -f2 -s gets only the second column. Often awk '{print $2}' is used instead. Awk can be very big. gawk is said to be big, and mawk is said to be minimal. cut seems more portable and smaller yet to me. -d sets the delimiter, -f2 is field two, -s is only print lines with the delimiter.

Try “set -x” before the command to see the different levels of shell script in the one line, do “set +x” after to get things back to normal.

Drew Daniels’ resume: http://www.boxheap.net/ddaniels/resume.html

September 20, 2009

Online vs Offline data storage

Filed under: Uncategorized — admin @ 1:31 pm

I have a ever rising need for data storage. Even if I stop accumulating data, or even cutting back what I store, I have to deal with failing media. I’ve had numerous hard drives die over the years, CDR’s have gone bad…

In my searches for cheap ways to store data, I came across some recommendations to use online storage. With Dreamhost offering 50GB for backups for free, this is something I’m going to start using. At $0.10 US/GB for overages, I hear it beats out quite a bit of competition. I’d prefer to stick with Dreamhost if possible as I know and use them.

Hard drive costs

So how much does hard drive storage cost, and how reliable is it? Well for the raw drive, I’ve recently been seeing about $1/GB CDN for SATA drives. My backup server is currently PATA with slow USB (likely version 1) so I’d likely need to spend some money upgrading the backup server. A cheap modern PC seems to go for about $300 CDN. I’d guess controller cards aren’t cheaper than $20.

Related costs

But the media cost isn’t the only cost. There’s also install time, maintenance time, and operation costs. One of the biggest operation costs at a house is electricity. Manitoba Hydro charges somewhere around $0.063/kwh CDN. That’s about 365.242199 * 24 * 0.063 = $552.246205 per kw-year.

Western Digital seems to say hard drives consume between 0.40 Watts to 5.4+5=10.4 Watts. I’ve seen elsewhere claiming 25 Watts for hard drives. So at the given kw-year, the cost of operation could be as low as about 0.40 * 552.246205 = $220 CDN. The more likely case of lets say 2 hours of operation, 1 hour idle to off, based on Western Digital’s “competing” write case of 5.4+4.5 watts in operation, 2.8 watts for idle, and 0.40 watts for standby is about
((((5.4 + 4.5) / 24) * 2) + (((2.8) / 24) * 1) + ((0.40 / 24) * 21)) * 552.246205 = $715.

Wow. I need to turn my computer off more often.

Simple comparisions

So lets take the best case for hard drives then and make it as big as seems reasonable right now. 1.5 TB is about 1,396.98386 GB. At Dream host, 3 years (typical hard drive warrenty) times 12 months times $0.10 times 1396 GB times 1.13 for padding for exchange rate, and possible volitility is about
3 * 12 * 0.10 * 1 396 * 1.13 = $5,679. Local storage would be $750 for electricty * 3 years, plus $150 for the hard drive, plus maintanence of lets say $250 is about (750 * 3) + 150 + 250 = $2,650.

So local storage is about half the cost in its best case. I’d say it’s more likely to be far less used, but at home I’d have to budget for expansion immediatly. Being at Dreamhost is good for a site disaster (like a house fire), but bad for access time.

Conclusion

More research is needed. If I had to make the decision tomorrow, I’d probably go with spending more on Dreamhost storage. I like the offsite feature, the low maintenance, and not having to spend as much time figuring out how much storage I might need.

Drew Scott Daniels’ resume

Older Posts »

Powered by WordPress