Posted on July 15th, 2009 in Development Practices | Permalink
To help get people started using the cucumber framework we presented tonight at the SF Ruby meetup. Here is the presentation and you can find all of the source code here:
http://github.com/howcast/cucumber_101_app/tree/master
Hope that everyone found it helpful and please feel free to ping us if you have any questions!
Posted on November 6th, 2008 in API | Permalink
Due to popular request we’ve now added Category support into our API.
If you haven’t already installed the Howcast Gem you can do so with: “sudo gem install howcast”
To get you up and running here is some sample code to interact with the Category taxonomy.
Posted on November 3rd, 2008 in Play | Permalink
What no posts about video transcoding?! Shocker, we know. We are always working hard to improve howcast.com, and despite what the droughts in blog posts may suggest, we actually do like to do things that do not involve work. Recently, I completed my first triathlon (the Santa Cruz Sentinel) which was a really fun experience. I am a complete hack, but for those of you that may be on the fence about giving it a go, I wrote a wiki guide that might help convince you to make the jump:
http://www.howcast.com/guides/2297-How-To-Not-So-Seriously-Train-For-a-Triathlon
Posted on October 27th, 2008 in Scalability, Video | Permalink
We recently had to transcode a large number of videos for the iPhone. Rather than try to scale out our in-house infrastructure, we thought it would be prudent to check out different projects people had been working on. One stellar example is Panda (whom our friend Ezra Zygmuntowicz at Engine Yard mentioned to us). Panda leverages Amazon’s S3, EC2, and Simple DB web services to offer a very flexible, cost effective, and scalable option for integrating video transcoding into any site. We’re likely going to do build something a bit more tailored for howcast.com, but will definitely try to see if there are things that we can contribute back to that project. We humbly tip our hats to the New Bamboo team for their fantastic work.
Posted on October 19th, 2008 in Video | Permalink
OK so we have admittedly just about worn out this topic, however, we do have some recipe improvements to share with you. We ran into some issues with our previous recipe for some videos on the iPhone. Here is our latest incantation which seems to work quite well:
ffmpeg -y -i [input file] -v 1 -threads auto -vcodec libx264 -b 300k -r 15 -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 6 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 768k -bufsize 2M -cmp 1 -s 480x320 -acodec libfaac -profile aac_low -ab 64k -ar 22050 -ac 2 -f mp4 [output file]
Note that you may need to adjust the resolution as appropriate. With respect to what changed, you will notice is that we lowered both the video and audio bit rate, are explicitly using the AAC low complexity audio profile (per Apple’s specs), and have also switched back to using stereo (vs. mono) for the audio output. You will need to ensure that you have a fairly recent version of FFmpeg installed with H.264 and AAC. We hope that you find this helpful and as always feel free to ping us with any questions.
Posted on August 28th, 2008 in Video | Permalink
A post or so ago we talked about podcast transcoding. Since then we’ve updated things a bit to support 640×480 video. The first thing that we did was update the FFmpeg recipe we use for transcoding:
ffmpeg -y -i [input file] -v 1 -threads auto -vcodec libx264 -b 500k -bt 175k -refs 1 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 6 -me_range 21 -chroma 1 -slice 2 -bf 0 -level 30 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -maxrate 768k -bufsize 2M -cmp 1 -s 640x480 -acodec libfaac -ab 96 -ar 48000 -ac 1 -f mp4 [output file]
This recipe was based on this Ubuntu community documentation. Note that we’re doing 1 pass versus 2 pass encoding and are using a single audio channel (still having interimittent audio issues if we use two).
The next challenge was setting the atom in the video such that the videos would sync properly from iTunes to video iPods and iPhones. As pointed out in Robert Swain’s post and also in an Apple forum thread, the development version of AtomicParsley has a feature to do this. We had to make some minor code tweaks to get things to compile properly (the changes were slightly different on OS X and Linux), but feel free to ping us if you get stuck. Once you have AtomicParsley compiled, here’s the command to run to set the atom:
AtomicParsley [output file] --DeepScan --iPod-uuid 1200 --overWrite
After that you should be set! Thanks to all of the contributors to these fantastic open source projects and let us know if you have any questions.
Posted on July 25th, 2008 in Tools | Permalink
Recently we migrated our source to git, and in the process became fans of github. We had been using Trac as our bug tracking software of choice, and it integrated smoothly with Subversion. Unfortunately, the git support was a bit lacking. There were a couple of plugins out there, but they didn’t really suit our needs. Fortunately, we found a better solution.
Enter Lighthouse. The guys at ActiveReload have designed a simple hosted issue and project management application that is easy on the eyes and plays great with github. We decided that a switch was in order. Since there’s currently no bulk import built into Lighthouse, we needed to migrate over a year’s worth of data. Luckily they offer an API and even a Ruby wrapper, which several folks have used to roll their own importers.
We first attempted to use Shay Arnett’s cool trachouse importer, but ran into some issues with our Trac authentication scheme and plugin setup. Trachouse fetches data from a Trac install by spidering your Trac website using hpricot. Since we were hosting our own Trac install, we figured it might be easier to just parse Trac’s sqlite database.
We put together a quick script to parse the trac.db and import it into Lighthouse using the lighthouse-api Ruby wrapper, and we (not so) cleverly dubbed it “Traclight”. We thought others might find this handy, and so there’s now a repo for it over at github.
To use it, you will need to edit the traclight.rb script and add your own conversion information, so the script will know how to translate Trac users, milestones, and states to the Lighthouse equivalents. You should create any user accounts and milestones ahead of time.
Next, export your trac database. See the trac FAQ for more information.
Now you are ready to run the script:
traclight.rb PATH_TO_TRAC_DB_FILE
There are a couple of caveats:
- Due to Lighthouse API limitations, ticket comments will be merged into a formatted description of the ticket.
- The tickets will show up as being reported by the owner of the account you specify in the ticket, not the original reporter. The original Trac reporter will show up in the description for the ticket.
- It is not possible to set the original date the ticket was reported in Lighthouse. The original reporting date of the Trac ticket will appear in the Lighthouse ticket description.
- Attachments are _not_ imported, as support currently doesn’t exist in the API.
- Starting with an empty Lighthouse project is recommended, since traclight will import the tickets in sequential order. That means that your ticket numbers should match up (as long as you never deleted tickets from Trac).
We used this to import around 5000 tickets spread over 15 milestones into Lighthouse using Traclight. Perhaps it’ll be of use to your team.
Posted on July 10th, 2008 in Video | Permalink
There are a plethora of ways to transcode your videos into a podcast-compatible format. We wanted to give you a quick walkthrough of how we use FFmpeg and AtomicParsley to do it.
First things first — check out Apple’s technical specifications to understand more about the podcast feed format and iPod-compatible video specifications. This post is going to focus on creating 320×240 videos in M4V format (which is really MPEG-4 video format) with a single audio channel. Why not 640×480? There’s some additional work that you have to do to update the M4V metadata so that the video will sync properly to video iPods and we’re still experimenting with it. Why a single channel of audio? Technically we should be able to do things in stereo (two channels of audio), but we ran into issues in production with the audio cutting out midway through some of our videos, and using a single channel of audio seemed to resolve the problem. You may not run into this issue so we’ll point out where to tweak the transcoding recipe so that you can use one or two channels.
The rest of this post assumes that you have the following software installed:
- FFmpeg - Open source video transcoding tool
- AtomicParsley - Open source tool for manipulating metadata in MPEG-4 videos
If you have a Mac and use MacPorts, here’s the easy/lazy way to do it:
sudo port install ffmpeg +lame +libogg +vorbis +faac +faad +xvid +libx264 +a52
sudo port install AtomicParsley
If you prefer to download and compile the packages, there are a number of useful resources to help guide you based on the operating system you are using — so we’re going to skip diving into additional detail there.
Step 1 - Transcode the video
We basically follow the recipe in Robert Swain’s post:
ffmpeg -i [input file] -f mp4 -acodec libfaac -ab 128k -ac 1 -vcodec libx264 -b 768k -flags +loop -cmp +chroma -partitions +parti4x4+partp8x8+partb8x8 -flags2 +mixed_refs -me umh -subq 5 -trellis 1 -refs 5 -coder 0 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -maxrate 768k -bufsize 2M -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 13 -s 320x240 -y [output file].m4v
A couple notes here:
- If you use a Mac and are on Tiger vs. Leopard, you’ll need to use
x264 versus libx264 for the -vcodec value
- Change
-ac 1 to -ac 2 if you want to try with 2 audio channels
Step 2 - Embed an image in the video’s metadata
Want a specific image to appear when your podcast videos display in coverflow?

Here’s where AtomicParsley works its magic.
AtomicParsley [output file].m4v --artwork [image] --overWrite
Note that per the Apple specs, you should use a 600×600 jpg or png image.
That’s all there is to it. Double-click on your M4V file and it should play in iTunes. Sync to your iPod and enjoy your video!
Posted on June 20th, 2008 in Scalability | Permalink
There are numerous options for performing background processing in Rails.
Here at Howcast, our method of choice is Backgroundjob (Bj).
“Backgroundjob (Bj) is a brain dead simple zero admin background priority queue for Rails. Bj is robust, platform independent (including windows), and supports internal or external management of the background runner process.”
Installing Bj
./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
./script/bj setup
This will create all the migrations you need to generate the job tables (note that there are also archive and configuration tables).
Distributing Bj
With Bj there is a persistent job queue in the database that workers can query to pick up pending jobs. This allows Bj to be easily distributed where workers can be run across various servers.
In your environment.rb simply add:
This will mean that the default Bj worker will not run on the web server. This means you can run various other workers on other servers/slices. Simply add this to the crontabs of the servers you want to distribute to.
Extending Bj
Although Bj is a full fledged solution for managing and running background processes, we needed some additional functionality on Howcast — dependency jobs and specialized workers. We needed some jobs to be run serially after one another and so we needed to specify a dependency job id that would need to be completed before the job submitted would be started. With this requirement also came another requirement of constraining specific workers to run specific types of jobs. To accomplish this we added a dependency_id to the bj_jobs table and added an option for running job works with specific tags (–only-tag parameter). This would allow you to run a worker with the following command:
This would cause this worker to only run jobs that were submitted with the ’specialized’ tag:
There is also an option to start a worker with –exclude-tag option to do the reverse of the example above.
With these two features combined you can create a pretty complex flow structure for jobs. Now you can split up larger jobs into smaller specialized jobs that specific workers will run in parallel or serially with the dependency_id set.
Installing the enhancements
Both these enhancements have been open sourced and are available here: http://github.com/howcast/backgroundjob/commits/dependencies_and_tags
We’ve found this to be an easy way to distribute background processing tasks and think some of you might find it to be useful as well.
Posted on June 20th, 2008 in Uncategorized | Permalink
At Howcast, we have a small engineering team that tries to accomplish a lot! Check out our about page to learn a bit more about who we are.