Experiments with Ramdisk

So after looking into RAMDISK as an awesome way to get hyper fast workspace, I began to realize what the issues were with modern day RAMDISK and why it is that it's not very popular.

First let's make a ramdisk. Easy enough, OS X is unix based, so let's just map some RAM off to a device. Then format and mount it. The process it outlined here:

http://osxdaily.com/2007/03/23/create-a-ram-disk-in-mac-os-x/

Next let's do a few speed tests. First, a standard SATA 7200RPM drive (seagate) nothing fancy, but no slow poke either.

Sure enough, the results are as expected. Sequential writes are decent, averaging out at about 35MB/sec. Random writes are weaker though, due to the mechanical nature of the drive.

Next let's take a look at the RAMDISK. We begin to start measuring things in GB/sec here. Scores of 1336 on an XBench test are pretty insane. Overall, just plain incredible.

So what can we use this in the real world?

1. High bandwidth video capture or rendering:

Sounds intriguing, as reading or writing to disk large HD files can be difficult on traditional HDDs.

2. Use as a photo storage while photoshopping

Would offer rapid access to all your images while loading.

3. Large file transfer over network etc.

If the bottleneck is the disk like I insist it is, then file transfers should be exactly 1.0GBps - overhead

So what happens really?

1. In video application, the problem lies within the size of the Ramdisk. While if you had a large capacity for storage of video captures or render, your'e still limited in terms of capacity. If you're working with a video that is smaller than 1-2GB in footage or render size, then chances are the render times won't be too terrible, that is, assuming you're not doing anything intensive. However most applications who are working with large intensive files are going to load them into RAM first anyway and then work with them, especially if you have the memory to spare that you're making ramdisks in the first place. So really, there's no advantage there.

2. Next, we have a similar roadblock here. Photoshop loads your images into RAM anyway so by loading all your work files into a ramdisk, then opening them, they might load quickly into the application, but you're better off ejecting the ramdisk and letting photoshop use the memory itself once everything is opened, which is really just a waste of time.

3. On the other hand, large file transfers seem to be quite good. I was able to transfer a 653MB TOD over SMB, gigabit, jumbo frames from jQuad ramdisk to MacBook ramdisk in 10 seconds flat. That's about 65.4MB/sec. That's about 0.5Gbps. I'm sure if I did some better tests with HTTP and other protocols, doing multiple tests I'd get a better result of .6 or .7, but overall it was quite impressive. In comparison, HDD to HDD did so in about 22.5 seconds, which is about 0.2Gbps. Pretty impressive increase.

But after considerable thought, it's virtual memory and intelligent applications that killed the ramdisk. Before the time of virtual memory, (and system 7 and 8

didn't have it as I recall) your system basically would have to use memory as carefully as possible, because once it was full, you basically were told to close applications and try whatever you wanted to do again. (If you want to experience such nostalgia, disable page file in Windows, it's a party and a half) but nowadays the operating system just writes out to disk any memory that isn't going to be in use (such as background applications) so that it can be given to the application at hand. Because applications can now use just about as much memory as they want, they do. Most applications now will simply load the entire file into RAM and work with it from there. As a result, it's really only the initial load into the application that the hard disk is penalizing your speed.

Now one of the other downsides of virtual memory and it's effect on ramdisk performance is that once another applications starts to request more memory that is currently available, the system will start to write memory to disk to offer it to the application. In this case, RAMdisk is not actually clear from this dumping to disk, and so if Photoshop decides to use more memory, then the system will simply write out your ramdisk the hard drive, and all of the sudden your hyper-fast temporary space is back on the disk and not so speedy anymore (that is, until it is restored to memory.)

All these factors contributed to the ultimate demise of ramdisk on the consumer desktop. While it's still very useful for nimble linux live discs and to my knowledge Google uses it extensively in their server technologies, ramdisk will most likely stay a benchmark dream rather than a useful tool for high performance computing.

Saturday, May 31st 2008 08:56 PM

Downtime Notice

Well it's been a good run folks. For over 345 days, the good old Dell Dimension 2200 has been whirring away in the basement of AHTR Headquarters, without a single glitch of downtime, but sadly all things must come to an end.

No, jole.ca is most definitely not going offline, just a brief time of downtime as AHTR reorganizes it's power and network structure, and while we're at it, replace jServer with a bit more modern equipment. (A Dell Optiplex to be precise, and more specs to follow as I receive more information)

Unfortunately, my efforts to convince Trent to delay the upgrade for a few weeks to that I can at least celebrate the one year anniversary of crappy dell desktop uptime, but he's a bit of a bogon and refuses to wait, so I guess I'll have to just have to suck it up and be happy with being 20 days short of my goal. (Dang!)

Moving forward, I can only hope that the new machine will be snappier and perhaps perform database querries a bit faster, but I kind of doubt there will be any noticeable difference.

I've considered a few options during this time, notably hosting myself on my own connection, or hosting with another company like dreamhost. Of course, dreamhost is no reliability expert, and if you really want reliable hosting you're going to have to pay for it. As for running it out of my house, I can sort of picture myself subsidizing the Internet bill to get a better connection into the house, but it doesn't quite compare to what I'm paying for colocation services right now, so I guess I'll put up with the yearly cable management sprees from trent.

The whole ordeal is going down at 10PM Saturday at last check-in, and with any luck he wont' fall asleep while doing it and we will be back online by the morning!

Saturday, March 01st 2008 01:03 AM

Flickrer 1.0, 300d uptime, and more

Well it's been a while since I've done...well...anything on the site.

It's been lots of happenings, and if you want to catch up on all that, listen to episode 6 of The Duck Podcast.

Today's a big day in the news though.

• Flickrer has now reached 1.0 release candidate 1 with full RSS/Atom support, as well as some major performance improvements thanks to some data caching measures.
• 300th day of uptime on jServer
• Apple announces New TV, Movie Rentals, MacBook Air, and more.

As for flickrer, I finally got around to doing some caching of the database files that it runs through EVERY time you load any page in the gallery, which I can't even imagine how hard it must have been on the servers for the larger galleries. This way, you MUST update manually or enable autoscan in the config in order for it to work. To rescan, just tack ?rescan=yes onto index.php and it will rescan. After that things should be really fast. If they aren't, make sure the gallery dir is writable.

Today is of course uptime 300. jServer has been putsing along for a mind blowing 300 days. Just a few months away from the 1 year mark!

And of course Apple has released some very long rumored products. My opinion of the whole MacBook Air is somewhat unimpressed. I was hoping for more of a professional product for the sub-notebook class, something with MORE I/O than the MacBook, rather than less. With no FireWire, no Line In, no ethernet, a SINGLE solitary USB 2.0 port, and a non-user replaceable battery, I find it a bit hard to imagine every switching to one of those guys, as beautiful machine as they are.

That being said, bravo on for Apple for engineering such a great machine. I think the price, comparing to Sony or other models is certainly a good value even if you are losing some I/O, and while Apple may be criticized for dropping just too much connectivity off of it's notebook, I think we all know that Apple is right about these things. If you look at the floppy drive among others, it's clear that they like to lead the pack in eliminating unnecessary bulk, and trim the fat, but I'm glad they didn't modify their existing line to do it this time.

As for movie rentals, I think it'll be a great hit for Apple, and that the Apple TV will take off for now. $20 is a ridiculous fee to pay for the new iTouch Apps, but I guess that's life. I hope the Apple TV UI comes to OS X though!

My money is probably going towards a new video camera, Sony DCRHC28 most likely, as it offers the best value for what I'm looking for. After that, I plan on building this machine:

Although I'll go for the Quad processor instead.

Tuesday, January 15th 2008 08:53 PM

iPod Improvements

So I don't think I've given a full review to my iPod Nano yet, but I will make a quick note on some cool features (a complete review is yet to come.)

One neat feature that I really like is the sleep timer. I regularly use my iPod to put me to sleep , and it's a great way to keep it's battery alive by the next morning. For those of you unfamiliar, it essentially turns off your iPod after a set time.

Now, by no stretch of the imagination is this a well designed feature. Something it is missing is: exact times, fade out at end of time (nothing like about to fall asleep and it CLIPS right off) but I did notice that if it isn't going to be able to finish the song in time, it won't start it, which is pretty neat.

I will complain however, that it no longer displays the time remaining on the playback screen like it did in previous generations.

One thing I never really thought of is just to use an On-The-Go playlist, let it play through, and then just let it put itself to sleep.

Either way, I though it was some neat functionality. Thanks again !

Monday, November 12th 2007 11:57 PM

Leopard Review

Well it's been about a day and a half with Leopard, and after reading a few reviews, I'm happy with it.

While it has it's little quirks, and Apple unnecessarily removed certain functionality for some arguably ineffective functionality, it still is a significant improvement over Tiger, and it's growing on me.

Overall there's no real grand features, but lots of polish, which is still really great. Most of the advancements occurred "Under the hood" at the API and Programming Level, which means while Leopard isn't SUPER fantastic right now, it will be soon as developers discover the great new technologies emerging.

I've compiled a list for things I hate and love about Leopard below.

Annoying things in Leopard:

• MDS still can't be turned off (MDS = spotlight service), although spotlight is a lot fastetr
• iTunes has skipped twice in the past half hour. It never does that. I think it might be the single-coredness of my first install.
• Stacks are a bit pointless, but fairly useful.
• Spaces are ok, but are more like switching around too much. I think it will come in handy for when I'm not in full desktop mode with dual display, but I can see using it with FCE and stuff
• Maudio doesn't work in any way (yet)
• The Menu Bar is a bit dark when there's a dark background, but it's less distracting, however there's a "star" right where the application is right now and it's driving me up the wall. For those of you dying for a hack for this, you can always just make 20 pixels along the top of your desktop background white (or black for a consistent silvery look) and you're done. No special apps or defaults writing required.
• Time machine shows a status bar all the time when it's backing up...maybe that'll go away, but it's ticking me off. Also, it didn't exactly work the first few times, but I think it is now.
• Notes folder randomly takes time to "Load" (Spinny thing) even though there's only one note, wtd? UPDATED: I hate Mail notes and Todos. I am not using them whatsoever.
• They did some weird thing with the bezel interface, making volume controls etc, harder to read. I'm pretty sure the drop shadow is gone.

Things I love in Leopard:

• Scrolling background windows
• The fact that when i'm done this note, I can just hit send. UPDATE: Not really, I went back to stickies.
• Time Machine is SOOOO awesome.
• Coverflow is neat, but not as useful as i thought it would be. We'll see. Ars Technica goes into details about the UI changes in the Finder, but I find it pretty good.
• QuickLook is excellent. It even does SVG! 
• FrontRow is better....I guess. I wish I could use the same keyboard shortcuts, and also, I hate that it's not the fwoosh it used to be. Also, it pauses iTunes when it launches. wtd is that? I don't like it.
• Instalations made with Installer now do the iTunes "Complete" sound when they're done. Also, an obnoxiously large green check mark shows up. Great.
• Mail.app is like Apple Outlook. I love it. Sort of.
• The new network preference pane are excellent.
• iCal, Mail, and pretty much everything is very quick. 
• Terminal updates are nice, but I still like iTerm
• My MacBook speakers actually have decent volume now UPDATE: Maybe not, I think iTunes was just preamped.
• AirPort finally shows secured/not secured in listing
• AirPort seemed to connect to networks really fast...
• Not only does "Scroll another window" work, but you can also right click a non active application, VERY useful for when say, you ahve a window on a second monitor, move your mouse over to the window, have something selected and right click to copy it. Amazingly, it works! AWESOME!!
• Cool feature: the checkbox in todo lists changes to an alert box when it is overdue
• Photo Booth effects are neat, the hacks are neater.

* Overall very solid.

Tuesday, October 30th 2007 08:02 PM