Custom APIs and Web Scraping for Science

So my team’s most recent application, Helix, involved genome visualization. We integrated it with the 23andme API, but still needed a way to find out interesting information about specific RSIDs (used by researchers and databases to refer to specific base pairs of DNA). By far the most useful and open source repository of genetic information is SNPedia, but I needed access to lots of information and to integrate calls to specific SNPs. Basically I needed an API. So being ever resourceful, I decided to make my own.

Tools for the task were an easy choice. I needed a small fast server that I could implement a web scrapper on. I have always wanted a reason to use BeautifulSoup, but it’s a Python library so I knew it would be easier to build a Python server to run the API endpoints. I chose Flask because of its lightweight nature and how much it reminds me of a Node/Express server at times.

Thankfully there are some really good tutorials for both Flask and BeautifulSoup, my favorites (and the ones I referenced when I hit weirdness) were Designing a RESTful API and Website Scraping with BeautifulSoup. Both of these tutorials said a lot of things better than I could have myself.

For access to my SNPedia API and information on how to use it, check out my project on GitHub.

Week 9: Highs and Lows and... WTF I only have three weeks left??

My week started out fairly average. We were all rolling along on our projects and then I noticed an event on the Hack Reactor Senior calendar. Tuesday, three weeks from this past Tuesday, is Hiring Day. Three weeks?? Not even now, more like two?? Oh, god. And yet, as much of a whirlwind as this has been and as often as I have impostor syndrome, I’m a little excited. I want to see what’s out there for me and find a job and learn and grow and do my instructors proud.

One slight stumbling block for me this week: Hacker in Residence positions. I applied and think I would have been accepted, but I had to bow out. After I sat down and thought about it, I just couldn’t justify being out of work that much longer (even on a stipend). It would have been fun to learn how to teach and spend some more time hacking on personal ideas, but that’s what weekends are for, right?

We also got to demo Helix for the first time. Helix is a gene visualization app that shows you your SNPs (base pairs) from 23andme that have traits attached to them (according to SNPedia.com). You can search traits or just browse your chromosomes for interesting info. It was built using a private beta framework (called Famo.us) that my team was lucky enough to get to be involved with. We have fingers crossed two more opportunities to demo Helix, one more run through at Hack Reactor and if all goes well, a private party/meetup for Famous.

Another fun thing that came out of Helix was that I got to dust off my Python knowledge. I had wanted to try BeautifulSoup (a Python web scraper) for a while now and I needed an easy way to pull rsid information from SNPedia so I created my own API wrapper! The code is available (including instructions on how to run it on your own) on my github account. It’s a tiny Python/Flask server that only has a couple of endpoints (the ones I really needed) but I’m thinking about expanding eventually.

And then I got sick. I came down with a cold on Friday and haven’t been to HackReactor since. I’ve been working from home, but mostly just trying to sleep, having weird dreams, and sounding pitiful. I’m getting better though and I will definitely be on-point on Monday to work out the last-minute details of Helix before all the demos come crashing around us.

Three more weeks until I graduate! My gift to myself - I’m attending the LAUNCH hackathon with two other women from HackReactor the weekend after it’s all over. I just don’t want to get lazy!

The Circus Waits to Takeoff

About a year ago for giggles I tried the Rails for Zombies class at Code School. I mean, it’s zombies. I am a huge zombie fangirl and I was getting back into Java programming at the time but wanted to learn something more web develop-y. Anyway, I got frustrated with Ruby/Rails. There were parts of it that just bugged me. So I stopped.

Then a few weeks ago I discovered Python. It was an immediate love. I started with Learn Python the Hard Way mixed in with Codecademy‘s Python lessons. Also I’m in love with the The Hitchhiker’s Guide to Python. I’ve been reading it like a novel.

Then I wandered around looking for things to do with my newfound love. Django just seemed too big for my very newb hackings. I can’t remember where but a blog somewhere out there in my searchings mentioned Flask. So I plugged that little baby into my cobbled together Python studio (i.e. my newly reformatted Windows 7 laptop) and felt a little overwhelmed. Undeterred I searched out tutorials and decided to start with this one which just creates some static pages (I’m moving on to more dynamic things as I post this).

Pic spam!!

My first Flask generated page. Isn’t it gorgeous?