github-dl: CLI for Downloading Gists and Repos

Problem

As part of my internship at Mozilla, I have been tasked with analyzing the structure of jupyter notebooks in the wild. On the pipeline/platform team, it is our responsibility to analysts and employees that the tooling that we provide is functioning well. In addition to other tools, analysts use jupyter notebooks and spark for analyzing large datasets and performing machine learning. In analyzing notebook usage across mozilla and outside of the organization, it is our goal is to create better templates for analysts to use and understand how we might be able to improve jupyter notebook.

Solution

In order to download or scrape notebooks from the web, I wrote github-dl to download notebooks. The tool suites my use case, because I needed to be able to filter and query for mozilla notebooks and additionally search for notebooks in a particular field of interest (ex: “spark” or “big data”).

github-dl is a command line wrapper on top of pygithub, gitpython, and requests. I used click in order to parse command line arguments and provide them to pygithub. pygithub performs the validation of credentials and the functionality behind searching for gists and repositories. GitHub API v3 does not allow querying for gists, so the only way in which a user can query gists.github.com is by passing usernames. Below is simple command line call to download my gists containing jupyter notebooks.

$ gist-dl cameres gist-notebooks --extension=ipynb

It’s much easier to perform querying on GitHub repositories, because GitHub API v3 allows passing a query like a user would on github.com. Below is a sample query that I wrote to download jupyter notebooks that are returned by searching for machine learning. I also include the parameter size to filter out repositories that could be massive. The destination of the repositories is github-notebooks.

$ github-dl 'machine learning language:jupyter-notebook size:<1000' github-notebooks

This tool can be used to download repositories and gists of any type. github-dl is available on the python package index (pip install github-dl) and has been tested for both python 2.7.12 and python 3.5.2. See the repository for more information.

My Favorite Music For Productivity

This post is entirely inspired by Thomas Frank’s recent series of posts on working with music and it will follow a similar structure. Because of the interest in the area of study/work music I decided to add to the conversation. I was originally interested in posting, because of Franks’ post Should You Listen to Music While Studying. There I found out about Theophany’s album Time’s End. He has since followed that post up with Can Music Be Designed to Enhance Your Brain’s Performance?. I highly recommend looking at both posts for more information on this topic (Thanks Thomas!).

Ever since I received my 5th Generation iPod Classic/Video on Christmas of 2005(?), I have always enjoyed listening to music while studying. Listed is a breakdown of study moods that I have and the artists that I listen to!

Energized Music

I used to do a lot of work with energized music, but found that it had been distracting me from the work that I had been doing. I’ve saved quite a lot of my energy music for time at the gym, as I find it conducive to hitting personal records, etc. However, I’ve been starting to spend time finding artists in different genres to add more variety to my library. I’ll often listen to this music when I’m…

  • programming personal projects
  • completing trivial tasks

Carpenter Brut

About two years ago, I started playing Hotline Miami and was extremely impressed by the soundtrack (and the game). Recently, I stumbled upon the album for Hotline Miami 2: Wrong Number and was really liked two artists in particular. Carpenter Brut is the first artist that I was first came across. I love the driving synths in each of his songs. Trilogy is my favorite album of his.

Mitch Murder

Mitch Murder is the other artist that I found through the album. I love Inceptor, Mitch’s top album, for many of the reasons that I love Trilogy. The album is packed with feel good 80’s inspired music.

Mellow Music

I’ve often listen to mellow music when I want to work through somewhat difficult exercises. I would include Theophany’s album, Time’s End, in this category, as well.

I’ll often listen to this music when I’m…

  • working on problem sets
  • writing a paper

C418

I’ve been listening to C418 since he released Minecraft - Volume Beta. He’s an independent composer and producer made fairly famous for his work in the game Minecraft. The albums that I enjoy include Volume Beta, one, and 72 Minutes of Fame. I also really like his new album, 148, but find it difficult to listen to while working.

Background Music

Background music is often required when I need to get in my zone, as Kanye and Jay-Z would say. Background music has been the most difficult for me to find, because while I like websites like Rainy Mood, I typically look for something a little more engaging. For a while, I tried to listen to ambient music, but I found that “new-age” piano is more effective for me. I’ll typically listen to this music an hour before I know that I’m going to bed or before mid-day, because of how relaxed I feel when I’m listening to it.

I’ll often listen to this music when I’m…

  • reading
  • editing a paper

Yiruma

Yiruma is a pianist and composer that primarily creates “new-age” works. These pieces are typically relaxing enough that they fall entirely in the background. Below are some of my favorite songs.

Ludovico Einaudi

Einaudi is yet another fantastic composer. I first started listening to Einaudi when I discovered his album Nightbook and instantly fell in love with many of his other albums. My other favorite albums are In A Time Lapse, Una mattina, and Elements. It is worth stating that, I didn’t find all of Einaudi’s albums suitable for studying. Albums like like Alexandria contain vocals that I found distracting.

Wrap Up

I hope you find each of these artists as helpful in your quest to study/work as I did! Send me a tweet if you have any suggestions for me! I’m always looking to listen to more music!

Disrupt You - My First Audible Book

Introduction

I was extremely excited to start reading Jay Samit’s Disrupt You, after I purchased the book on my 22nd birthday. This was my first book on Audible and certainly not my last. I was first interested in the book, because of strong reviews that the book received on Amazon and Audible. However, I didn’t imagine that the book would be nearly as applicable to my life, as it was. After reading the book over the last 13 days, I am pleased to say that I can’t wait to reread this book later.

Disrupt You Cover

Overall Review

If you’re a potential reader, I strongly recommend at least taking a looking inside the book on. It’s free to do on both Amazon and Audible. Within the introduction, Jay presents many of the case studies that he will later discuss throughout the book. Jay also outline his personal journey from a “20 person startup to being a global officer of an international company with over 160,000 employees.” Jay also addresses his opportunities at Universal, EMI, and Sony, that later read extraordinarily well. Each case study mentioned in the book is unique and doesn’t serve to fluff the book with information.

However, this book isn’t just for those interested in entrepreneurship or business ventures. It’s about addresses how today’s greats have conquered problems and created massive opportunities. The process of disruption is “entering the fray” as Jay so eloquently says. If you’re interested in separating yourself from your peers and asking more from life, this book will serve as a great resource.

Jay does touch on rather technical subjects such as cloud computing, wearable technology, 3D-printing, and the internet of things, but in a manor that is accessible to people that might not be as nerdy as me. He has also organized this book in a manor such that this material falls toward the back. This has worked well to introduce topics like self-disruption, that can be applied by a very broad population, while the technological advances in production and computing may not interest the everyday reader.

Jay goes on to talk that about how these technologies disrupt industries that they are not even immediately connected to. He also talks about how they too can create new opportunity. I think while the average reader may not be interested in all of the topics, it is important to know about the changes in the future and how you can be part of the disruption.

Overall Recommendation

Disrupt You, while the first audio books that I have listened to, will likely hold the position of my favorite book for quite some time to come. “Disrupt You, is for anyone that is looking for a promotion a raise or more satisfaction in their lives”, Jay mentions at the end of the first chapter. I can strongly say that Jay’s book has strongly affected both my professional goals and personal goals for the future. I highly recommend it to anyone.