FLIP Your Animations

Instead of animating “straight ahead” and potentially doing expensive calculations on every single frame precalculate the animation dynamically and let it play out cheaply.

source-map-explorer

The source map explorer determines which file each byte in your minified JS came from. It shows you a treemap visualization to help you debug where all the code is coming from.

Rewind

Rewind efficiently and continuously records your Mac's screen so at any moment you can rewind to see an "instant replay" movie of the last 60 seconds of what was shown on screen.

DataLoader

DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various key-value store backends and reduce requests to those back-ends via batching and caching.

Unicorn

Unicorn is a lightweight multi-platform, multi-architecture CPU emulator framework.

Adding edge swipe to SFSafariViewController

Subclass SFSafariViewController, add a view on the edge, and add a UIScreenEdgePanGestureRecognizer to it.

Aspects

Library for aspect oriented programming. Think of Aspects as method swizzling on steroids. It allows you to add code to existing methods per class or per instance, whilst thinking of the insertion point e.g. before/instead/after. Aspects automatically deals with calling super and is easier to use than regular method swizzling.

Flexbugs

A community-curated list of flexbox issues and cross-browser workarounds for them.

velocity-react

React components for interacting with the Velocity DOM animation library.

BBEdit Expert Preferences

There are a number of behaviors in BBEdit for which there are no UI controls in the Preferences window; this is typically because the settings are so obscure that placing them in the Preferences window would just make it complicated. In fact, some of these settings were in the Preferences window, and have been removed in order to simplify the interface; in such cases, any changes you made previously will be honored, even though the UI in the Preferences window is gone.

asciimoo/memspector

Inspect memory usage of Python functions.

facebook/pfff

pfff is a set of tools and APIs to perform some static analysis (e.g. to find bugs), dynamic analysis, source code indexing, code search, code visualizations, code navigations, or style-preserving source-to-source transformations such as refactorings on source code. There is good support for C, Java, PHP, JavaScript, HTML, CSS, and preliminary support for C++, Rust, C#, Erlang, Lisp, Scheme, Haskell, Python, OPA, Sql, and even TeX. There is also very good support for OCaml and noweb (literate programming) so that pfff can be used on the code of pfff itself.

Bridgy

Bridgy is a service that pulls comments, likes, and reshares on social networks back to your web site. You can also use it to post to social networks - or comment, like, reshare, or even RSVP - from your own web site.

TIS-100

TIS-100 is an open-ended programming game by Zachtronics, the creators of SpaceChem and Infinifactory, in which you rewrite corrupted code segments to repair the TIS-100 and unlock its secrets. It’s the assembly language programming game you never asked for!

react-strict-prop-types

A higher order component that raises an error if a component is used with an unknown property. A property is considered unknown when it is not defined in the component propTypes declaration.

Cosmos

Cosmos scans your project for React component fixtures and loads them inside ComponentPlayground, enabling you to:

1. Render your components under any combination of props and state
2. See component states evolve in real-time as you interact with running instances

Six Speed

Performance of ES6 features (as implemented natively and via transpilers) relative to the ES5 baseline operations per second.

v8-perf

Notes and resources related to v8 and thus Node.js performance.

Delivering high scroll performance

To diagnose jerky scrolling in their iOS app, Facebook measures the frame rate and when it drops below a certain threshold it has a watchdog thread that sends a signal to the main thread to get a stack trace of what is resulting in slow rendering. Sample code provided.

@import Frameworks into LLDB

Can now say expr @import Foundation(or UIKit or Cocoa) to get access to Foundation/UIKit/AppKit methods in the Xcode debugger.

jscodeshift

jscodeshift is a toolkit for running codemods over multiple JS files.

See also js-codemod, which contains a collection of codemod scripts based on jscodeshift.

ngrid

ngrid is a tool for interactive browsing large tabular datasets in a text-based terminal enviroment. ngrid is to tables as less is to text.

Otto

A JavaScript interpreter in Go.

Favicon Generator

Generate the favicon pictures and HTML code that work on all major browsers and platforms.

lz-string

JS compression library meant to make storage of data in quota-limited localStorage more manageable.

ramjet

Morph DOM elements from one state to another with smooth transitions.

ClangFormat

ClangFormat describes a set of tools that are built on top of LibFormat. It can support your workflow in a variety of ways including a standalone tool and editor integrations.

foursquare/FSQCellManifest

A UITableView and UICollectionView delegate and datasource that provides a simpler unified interface for describing your sections and cells.

A Complete Guide to Flexbox

paulirish/memory-stats.js

Gauge for JS heap size. See also stats.js for an FPS display.

Detecting Element Resizing

sdecima/javascript-detect-element-resize and KyleAMathews/element-resize-event both appear to use two <div> and scroll events to detect resizing without polling.

Animating SVGs With CSS Transforms

Necessary for hitting 60 FPS on mobile.

NuclearMail

NuclearMail is an experiment of writing a webmail client using React and the Flux architecture. It runs completely in the browser and uses the Gmail REST API.

UIKit Debug Mode

UIKit has hidden, useful debug flags for gesture/touch/animation logs, enable "InternalBuild" mode to switch them on.

fishhook

fishhook is a very simple library that enables dynamically rebinding symbols in Mach-O binaries running on iOS in the simulator and on device. This provides functionality that is similar to using DYLD_INTERPOSE on OS X. At Facebook, we've found it useful as a way to hook calls in libSystem for debugging/tracing purposes (for example, auditing for double-close issues with file descriptors).

Towards declarative touch interactions Slalom

A system with prototype implementation for concisely describing and implementing a class of touch interactions.

Xcode Consolation

Tips on using the Xcode debugging console (i.e. LLDB), notably on using regular expressions to add lots of breakpoints to trace execution.

Hopper+lldb for iOS Developers: A Gentle Introduction

Tutorial on how to use Hopper to find out how Apple frameworks work.

HeapInspector

HeapInspector is an iOS debug tool that monitors the memory heap in your app. You can discover memory leaks, no longer needed living objects and more issues directly on your device without ever starting Instruments.

Quill - A Rich Text Editor with an API

ContentEditable-based extensible rich text editor

widged/kinetic

Kinetic Scrolling with JavaScript

Immutable Data Collections

Immutable data cannot be changed once created, leading to much simpler application development, no defensive copying, and enabling advanced memoization techniques.

Immutable provides List, Stack, Map, OrderedMap, and Set by using persistent hash maps tries and vector tries as popularized by Clojure and Scala. They achieve efficiency on modern JavaScript VMs by using structural sharing and minimizing the need to copy or cache data.

Immutable also provides a lazy Seq, allowing efficient chaining of collection methods like map and filter without creating intermediate representations. Create some Seq with Range and Repeat.

FrameViewer How-To

Good overview of about:tracing and of Chrome's rendering pipeline in general.

scribe/BROWSERINCONSISTENCIES.md

contentEditable browser inconsistencies that The Guardian ran into while developing their Scribe library.

GitHub Cheat Sheet

A collection of cool hidden and not so hidden features of Git and GitHub.

facebook/chisel

Chisel is a collection of LLDB commands to assist in the debugging of iOS apps.

Lesser-Known JavaScript Debugging Techniques

debug(), monitor(), monitorEvents() and a Chrome extension to extend these to native methods and custom events.

ImageFiller

Quickly fills your iPhones Photo Album with thousands of random images from lorempixel.com.

LICEcap

LICEcap can capture an area of your desktop and save it directly to .GIF (for viewing in web browsers, etc) or .LCF (see below).

Sort of related is Sublime Text's hack to animate screenshots.

Clipping Magic

Instantly create masks, cutouts, and clipping paths.

BBUDebuggerTuckAway

Xcode plugin for auto-hiding the debugger once you start typing in the source code editor.

JavaScript Memory Profiling

Deep dive into the Chrome/V8 heap profiler.

fseventer

Observes filesystem changes using the same underlying API as Spotlight
and provides a graphical representation of the file activity.

MachOView

MachOView is a visual Mach-O file browser. It provides a complete solution for exploring and in-place editing Intel and ARM binaries.

Library Inspector

Library Inspector is a simple tool for inspecting the contents of compiled object code, archives, dynamic libraries (dylib), frameworks and applications. There are two main components, the Library Inspector application, and the Library Inspector Quick Look plugin. The main features provided by Library Inspector include:

• Quickly display local and exported symbols using colors to visually represent the type of symbol.
• Flexible filtering search for symbol name and type.
• Dump Objective-C libraries to header files quickly and easily.
• Disassemble compiled code to symbolic assembly language at the click of a button.
• Library-specific icons generated using Quick Look including the architectures included in the library.

Hopper

Hopper is a reverse engineering tool for OS X, Linux and Windows, that lets you disassemble, decompile and debug (OS X only) your 32/64bits Intel Mac, Windows and iOS (ARM) executables.

Avoiding repaints due to :hover selectors while scrolling

...by disabling pointer events during the scroll.

Reverse engineering Xcode with dtrace

Sample dtrace scripts for tracing Objective-C message sends (with filtering and indentation to make flows easier to follow).

paulirish/break-on-access

Simple Dev Tools snippet for adding breakpoints when JavaScript changes property values (including ones for built-in objects).

Python Memory Leaks

Object graph visualization library, as described in these blog posts:

• Hunting memory leaks in Python
• Python object graphs
• Object graphs with graphviz
• Tracing Python memory leaks

On Mac OS X graphviz should be installed to get the dot tool.

SimFinger

Helper tools (fake apps, fake carrier, tap indicator) for recording iOS app screencasts. See this blog post for more details.

PonyDebugger

PonyDebugger is a remote debugging toolset. It is a client library and gateway server combination that uses Chrome Developer Tools on your browser to debug your iOS application's network traffic and managed object contexts.

Clear in the iCloud

Clear uses a custom system built on top of iCloud File Storage and it works in a similar fashion to Operational Transformation. The post proceeds to cover the reasons for choosing iCloud, then explores iCCD and subsequently builds a synchronization system from the ground up.

NS: Poor Man’s Namespacing for Objective-C

Preprocessor hack that works around the lack of namespaces in Objective C.

Alternative UIWebView to Objective-C communication paths

WebViewJavascriptBridge (and similar libraries) rely on loading a custom URL scheme in an iframe and intercepting the request in shouldStartLoadWithRequest. Other approaches include embedded WebSockets and HTTP servers (see LinkedIn.

One alternative to consider is modifying document.cookie for some magic cookie name value, and then listening for those changes on the iOS side via NSHTTPCookieManagerCookiesChangedNotification. That might have fewer side-effects that triggering a (sub-)frame load.

Tuneup JS

Tune-up is a collection of JavaScript utilities that builds upon and improves the UIAutomation library provided by Apple for testing iOS applications via Instruments. While the JavaScript library provided by Apple is fairly complete and robust, it is a bit wordy and repetitive at times. This project aims to reduce as much pain as possible when writing UI tests by enhancing the existing API. It also provides some basic structure for your tests in a manner familiar to most testers.

JavaScript Madness: Keyboard Events

Summarizes the results of some browser tests done while attempting to implement key stroke handling code in JavaScript. It documents inconsistencies in the way different browsers implement keyboard events.

CSS Table Border Conflict Resolution

Explains why this works.

iOS WebKit modifications

Apple does code dumps on their open source releases site. For iOS, this contains WebCore and JavaScriptCore (because of LGPL), but not WebKit (BSD).

iOS 6.1 seems to track the open source WebKit repository up to r116247. You can diff Apple's code against the open source repo at that revision to see what is different in iOS.

MiniXcode

Plugin that makes it easier to run Xcode without the main toolbar. It adds keyboard shortcuts for selecting the active scheme and device (Ctrl+7 / Ctrl + 8), and a compact popup menu in the window title bar that shows the currently selected run configuration.

FastClick

FastClick is a simple, easy-to-use library for eliminating the 300ms delay between a physical tap and the firing of a click event on mobile browsers. The aim is to make your application feel less laggy and more responsive while avoiding any interference with your current logic.

See also Google's fast buttons article. Though it may not be necessary anymore, at least on Android (stock browser doesn't do a delay for fixed viewport sites since Gingerbread, and neither does Chrome).

Deep JS object comparisons

Micro-benchmark comparing doing manual object property traversal and using JSON.stringify and string comparison of the result. One might think that the JSON approach would be spending more time in native code, thus would be faster, despite the string allocation overhead. But the manual property traversal is always faster (JIT FTW).

Array loop iteration style

So long, for (var i = 0, p; p = a[i]; i++) {...}. Also related are these tests of Array.prototype.splice() performance.

Screenshot Uploader App

Minimal iOS app that uploads screenshots to a designated folder in Dropbox. Once configured, it should just be a matter of briefly launching it once the screenshot is taken (background HTTP completion service can be used to finish the upload).

Automating iOS Screenshots

Post describing the use of a self-driving screenshot mode for apps (uses WaxSim for a CLI interface to the simulator). See also the original author's take.

UI Screen Shooter is a variant that uses UI Automation.

On the desktop, the screencapture tool may be useful.

View chunk-level diffs of PNG files

1. Download zlib and build it with ./configure and make.
2. Download pngcheck.
3. Make sure that the zlib directory is called zlib and is a sibling of of the pngcheck directory.
4. Build pngcheck with make -f Makefile.unx.
5. Put pngcheck somewhere in your path.
6. Create a pngdiff shell script somewhere in your path:

   #!/bin/bash
   
   LOCAL_FILE=`mktemp /tmp/pngdiff.local.XXXXXX` || exit 1
   pngcheck -v $1 > $LOCAL_FILE
   
   REMOTE_FILE=`mktemp /tmp/pngdiff.remote.XXXXXX` || exit 1
   pngcheck -v $2 > $REMOTE_FILE
   
   bbdiff --wait --resume $LOCAL_FILE $REMOTE_FILE

   (replacing bbdiff with your preferred text diffing tool)
7. Add the following lines to ~/.gitconfig:

   [difftool "pngdiff"]
     cmd = pngdiff "$REMOTE" "$LOCAL"

8. Run git difftool -t pngdiff to view diffs.

Relatedly, to view diffs of images in Preview.app (using the up/down arrows to toggle between the old and new versions), add this to ~/.gitconfig:

[difftool "imagediff"]
  cmd = open -n -W -a preview "$REMOTE" "$LOCAL"

Clutch.io

Clutch.io is two projects:

• Native A/B testing for iOS and Android
• A toolkit for developing hybrid native/HTML applications for iOS

Animated GIFs the Hard Way

Python development in Pants

Pants makes the manipulation and distribution of hermetically sealed Python environments painless.

Word wrapping/hyphenation using CSS

iOS Runtime Headers

iOS Objective-C headers as derived from runtime introspection. Generated using RuntimeBrowser.

ssh-copy-id

ssh-copy-id installs your public key in a remote machine's authorized_keys. It should be in the contrib of the OpenSSH repository. There are other mirrors.

GPU Accelerated Compositing in Chrome

Background and details on the implementation of hardware-accelerated compositing in Chrome.

Learning Modern 3D Graphics Programming

Chromium Setup

Links:

• Chromium SVN Credentials
• Chromium Git Setup
• Mac build instructions
• Setting up Ninja
• Some things in the faster Linux builds page apply to the Mac too (disable NaCl, use shared libraries)

Recipe:

1. Visit Chromium Access and get credentials.
2. Run svn ls --username mihaip@chromium.org svn://svn.chromium.org/chrome/trunk/src and svn ls --username mihaip@chromium.org svn://svn.chromium.org/blink/trunk to verify said credentials.
3. cd ~/Developer/tools/
4. svn co http://src.chromium.org/chrome/trunk/tools/depot_tools
5. cd ../source
6. mkdir chromium
7. add ~/Developer/source/chromium to the list of paths that Spotlight excludes
8. cd chromium
9. fetch blink
10. cd src
11. rm -rf out/Debug
12. export GYP_GENERATORS="ninja"
13. export GYP_DEFINES="component=shared_library disable_nacl=1"
14. ./build/gyp_chromium
15. ninja -C out/Debug chrome

Updating:

1. cd ~/Developer/source/chromium/src
2. git pull --rebase origin master
3. export GYP_GENERATORS="ninja"
4. export GYP_DEFINES="component=shared_library disable_nacl=1"
5. gclient sync
6. ninja -C out/Debug chrome

Mailing lists that mihaip@chromium.org was on

If I ever want to re-join:

• apps-dev
• chromium-apps
• chromium-apps-reviews
• chromium-dev
• chromium-extensions
• chromium-html5
• size-reduction
• webkit-dev
• public-webapps
• public-sysapps
• public-native-web-apps
• whatwg-whatwg.org
• www-dom

Disable +1 buttons in Google Reader

Reload Reader or mark an item as read (so that the command token gets refreshed) and then run the following in the DevTools console:

var xhr = new XMLHttpRequest();
xhr.onload = function() {
  console.log(xhr.status);
};
xhr.onerror = function() {
  console.log('error');
};

var params = '';
function addParam(name, value) {
  if (params) {
    params += '&';
  }
  params += encodeURIComponent(name) + '=' + encodeURIComponent(value);
}

addParam('T', _COMMAND_TOKEN);
addParam('k', 'item-actions');
addParam('v', '{"plusone-action": false, "share-action": true, "email-action": true, "tags-action": true}');

xhr.open('POST', '/reader/api/0/preference/set');
xhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
xhr.send(params);

Once you see 200 printed, you can reload Reader, and the +1 buttons should be gone.

To reset things back to the way they were, change the 'v' value to '{}'.

Developer and Power Users Tool List for Mac OS X

Vector Art in Photoshop

• Vector shapes in Photoshop CS6
• Helpful actions
• Aligning pasted vectors

Xcode Tips

Xcode 4 splits, Adding a delay to auto-completion and Chromium's tips.

Deploying at GitHub

Deploying is a big part of the lives of most GitHub employees. We don't have a release manager and there are no set weekly deploys. Developers and designers are responsible for shipping new stuff themselves as soon as it's ready. This means that deploying needs to be as smooth and safe a process as possible.

The best system we've found so far to provide this flexibility is to have people deploy branches. Changes never get merged to master until they have been verified to work in production from a branch. This means that master is always stable; a safe point that we can roll back to if there's a problem.

codesearch - Indexed regular expression-based search

Code Search is a tool for indexing and then performing regular expression searches over large bodies of source code.

Status Boards

Panic's and Cultured Code's.

Hybrid Mobile Apps

Tumblr for iOS: Uses a JSON API and Mustache templates like a mobile site would, but does template evaluation on the native side, with the UIWebView just being used to render the resulting HTML. This makes the absence of a JIT less critical.

LinkedIn: Works around the lack of a good web/native bridge by embedding an extremely lightweight HTTP server in the application itself. The HTTP server exposes native functionality through a very simple REST API that can be consumed in the embedded HTML controls through standard JavaScript HTTP requests.

See also this SXSW panel.

Update on 12/7/2012: Tumblr appears to have changed to being purely native.

Commute Tracker iPhone App

iPhone app that automatically keeps track of your commute and helps you detect patterns. Using geo-fencing, it could keep track of when you leave your designed home/work spots, and then do a continuous GPS track along the way. Once data is gathered, it could be visualized (e.g. by day of week, or by departure time), displayed on a map to show alternate route rates (e.g. Grant Road vs. 85) and with enough smarts, turned into a (local) push notification that tells you which way to go today.

Commute Tracker gets pretty close to this (it's automatic), but it's buggy (I had a 1 second commute), tracks too much (I want to exclude days where I make a grocery store detour) and is on the ugly side. Daily Commute is prettier, but it's too manual, requiring commutes to be started and stopped.

Feed filtering Stream Spigot tool

Build a Stream Spigot tool for filtering of feeds. Unlike Feed Rinse or Yahoo! Pipes, it wouldn't involve taking a feed URL, giving it to the tool, and getting a filtered URL to subscribe to in return. Instead, it would work by connecting to your Google Reader account and then marking matching items as read, so that they don't show up (if using new items only view).

This would have the advantage of not having to re-subscribe when deciding to filter something, benefiting from the regular crawl speed (vs. filtered feed URLs generally having only one subscriber and thus being in the slow crawl bucket) and having item metadata be preserved (though that matters less now that likes and shares are gone).

The filtering could either be done periodically, or for feeds that are PubSubHubbub-enabled, it could be triggered by PSHB pings. It does mean that there is a window during which items that should be filtered our are visible, but the filtering is likely to be cheap enough (get items since the last check, apply a read tag) that it can be done quite frequently.

In addition to the read tag, a "filtered" state tag should also be applied to the items, so that these items can be differentiated during subsequent API calls.

App Reload Helper Chrome Extension

To make the Chrome Apps edit-reload cycle faster (especially when there's a build/compilation step), a helper extension could be written. It could reload the app using the management API. More importantly, it could have a custom hook to trigger the re-build/compile step before reloading the app. This could be as simple as an HTTP request to a local server, or something fancier with native messaging once that's available. This could all be triggered via a keyboard command using the commands API.

Chrome Extension Reloader Macro is something along these lines, albeit targeted at extension developers. LiveReload could also be extended to support this use case.

Code Explorer (Packaged) App

Turn Source Quicklinks into a packaged app. It can use <browser> to embed the Chromium code search and have a custom toolbar that allows switching to SVN history, blame, etc. It can also log recently pulled up docs in a sidebar, support looking up files by name.

Credit card reader Chrome extension

The Square card reader converts the magnetic stripe on a credit card to audio. Using the getUserMedia() API from WebRTC, it should be possible to get at that audio data (via WebAudio). mslib is a library for decoding this magnetic stripe data (blog posts with more details). It could be incorporate via NaCl. Rhombus is another project that does this (albeit Java-based).

One possible application is in a Chrome extension that implements payment form filling out by swiping a card.

Atop

Atop is an ASCII full-screen performance monitor that is capable of reporting the activity of all processes (even if processes have finished during the interval), daily logging of system and process activity for long-term analysis, highlighting overloaded system resources by using colors, etc. At regular intervals, it shows system-level activity related to the CPU, memory, swap, disks, and network layers, and for every active process it shows the CPU utilization, the memory growth, priority, username, state, and exit code.

What are some time-saving tips that every Linux user should know? - Quora

What are some time-saving tips that every Linux user should know?

Git BBEdit Package

A BBEdit package to make working with Git easier.

Foundation Icons Fonts

Custom icon sets that are stored in a handy web font.

Top 10 DTrace scripts for Mac OS X

Mosh

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.

Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Lights Out

iPhone app that lets you put nearby Macs to sleep (discovery can be handled via Bonjour). Unlike iShutdown or Control!, it shouldn't require installing a special daemon on the Mac that's to be controlled. iSleep claims to do this via Remote Login (i.e. ssh), but the reviews seem mixed. If remote Apple events are still possible, then it should be feasible via that. If not (or as an alternative), whatever mechanism Apple Remote Desktop uses could be used here too.

apt-file and finding Ubuntu packages

apt-file can find which packages include a specific file. Ubuntu Packages Search also provides a web UI.

Freeze inactive background Chrome tabs

In the vein of the Firefox extension Dormancy, build a Chrome extension that frees up resources used by background tabs that haven't been accessed in a while. Using the chrome.tabs.onUpdated event, an LRU list could be maintained. Inactive tabs would have a screenshot taken via chrome.tabs.captureVisibleTab, and the tab would then be navigated to a chrome-extension:// URL that shows that screenshot (thus making all inactive tabs live in the extension's process, instead of separate renderer ones). When the tab is re-selected, history.back() could be used to back to the live page (by using history traversal, form/scroll position restore is invoked, retaining as much of the state as possible).

Optionally, there could be a browser action that displays the total/live/frozen tab count, in the same way that Tab Count does.

Derby

MVC framework making it easy to write realtime, collaborative applications that run in both Node.js and browsers.

Pretty-print inline diffs in emails

Rietveld (and presumably other code review tools) inline diffs in emails. Even if they don't, they generally have a URL that points at the review. A Gmail gadget could be written that inlines the diffs (if not already inline), and pretty-prints them (with syntax highlighting).

GitHubDiff does something like this for GitHub emails.

TodoMVC

TodoMVC is a project which offers the same Todo application implemented using MV* concepts in most of the popular JavaScript MV* frameworks of today.

Solutions look and feel the same, have a common simple feature-set and make it easy for you to compare the syntax and structure of different frameworks so you can select the one you feel the most comfortable with.

Aleph

Google has shut down Code Search and the Social Graph API. Replicating those services outside of Google is hard; they presuppose a scalable, well-oiled crawling/indexing/serving machine. Only Google has been able to release interesting n-gram data (from books too).

Along the same lines, the HTTP Archive tracks 55K URLs (with a goal of 1M by the end of 2012). W3Techs also only looks at the Alexa top 1 million. Presumably this is because crawling is hard. If Foursquare wanted to know how many pages have a link to a foursquare.com URL, would they want to crawl the whole web?

"Aleph" would be an infrastructure company that offers a few services:

1. Large (billion and up) archive of sites on the internet. This would include not just raw HTML, but the full set of resources necessary to recreate that page (think crawling with headless WebKit and capturing all HTTP requests).
2. Ability to do quick (semi-interactive) analyses over the crawl data that has been pre-processed (think regular expressions, or Dremel-like queries for attributes).
3. Ability to run arbitrary MapReduces over the data (see ZeroVM for a way to do this safely).
4. Ability to import new datasets (whether web-like or not)

For the web index to be useful for some applications, it would need to have a PageRank-like attribute per page to expose importance (and/or a number of visits/traffic).

In theory the Internet Archive has a lot of this data. They make their data available "at no cost to researchers, historians, and scholars," thus their willingness to license is unclear.

As for the name, the Aleph was a MacGuffin-like device from Mona Lisa Overdrive that a large copy of the Internet stored.

Update on 5/7/2012: Blekko appears to offer basic grepping functionality.

Update on 2/24/2013: CommonCrawl is a non-profit with 6 billion pages indexed and available for MapReduce.

The Locker Project

A Locker is a container for personal data, which gives the owner the ability to control how it's protected and shared. It retrieves and consolidates data from multiple sources, to create a single collection of the things you see and do online: the photos you take, the places you visit, the links you share, contact details for the people you communicate with, and much more. It also provides flexible APIs for developers to build rich applications with access to all of this information.

The Locker Project is an open source development effort, permissively licensed, and sponsored by Singly. Singly provides a personal data service based on this technology, and also welcomes its use by anyone for any purpose.

MVC Framework for Node — Matador

Matador is a clean, organized framework for Node.js architected to suit MVC enthusiasts. It gives you a well-defined development environment with flexible routing, easy controller mappings, and basic request filtering.

krillapps

CodeRunner: Edit and run code in any programming language with just a single click. Write and test code, run scripts, work with algorithms, or simply experiment with a new coding or scripting language.

Patterns: A simple yet powerful tool for working with regular expressions. Build great patterns quickly and effortlessly with syntax coloring and with matching and replacing occurring in real time.

Building Games with Web Tools

Code

Editors

1. Cloud9 IDE

Resources

1. jswiki
2. microJS

---

Sound

Music

1. Noteflight (brief tutorial on how to write music for games)
2. Sonant Live, a tracker
3. Aviary Roc, beat sequencer

Editing

1. Aviary Myna

SFX

1. jsfxr

---

Graphics

Pixel editors

1. Tzigla
2. Pixie
3. Piq
4. Draw!
5. Make Pixel Art
6. TinySprite

Voxel editors

1. Cubescape
2. QBlock

Vector graphics

1. svg-edit
2. Aviary Raven

Painting

1. Muro
2. Sketchpad
3. Aviary Phoenix
4. Pixlr

Procedural

1. Pinello (requires facebook login)
2. Recursion Toy
3. Algorithm Ink

Misc

1. Aviary Peacock, effects/filter editor
2. Aviary Toucan, palette creation tool
3. COLOURlovers, palette/color discovery
4. Adobe Kuler, palette/color discovery

dpup/surface - GitHub

Surface is a JS library intended to help manage single-page applications (or "Ajaxy" apps). It promotes decoupling of components and provides tools for navigating between "screens" and integrating with the browser history.

PhantomJS and confess.js

PhantomJS provides a headless, yet fully-featured, WebKit browser that can easily be launched off the command-line, and then scripted and manipulated with JavaScript.

confess.js is a small library that uses PhantonJS to make it easy to analyze web pages and apps for various purposes.

benvanik/node-airplay

node-airplay is a client library for Apple's AirPlay remote playback protocol. It implements a simple AirPlay device browser using mdns and command interface.

CodeKit

CodeKit automatically compiles Less, Sass, Stylus & CoffeeScript files. It effortlessly combines, minifies and error-checks Javascript. It optimizes images, auto-reloads your browser and lets you use the same files across many projects.

Ugorji Nwoke: Dev Tool for GO AppEngine

Alternative development runtime/SDK for Go App Engine that relies less on the Python SDK.

Responsive IMGs

Part 1, Part 2, and Part 3 on images delivered using the HTML IMG tag that come from different sources depending the screen size. See also this post on media queries.

UILayer

UILayer provides a JavaScript API on top of WebKit for working with the concept of layers. Instead of manipulating DOM elements using a myriad of mixed concepts, you go though a single, well defined API.

glfx.js

Adjust photos in your browser in realtime with glfx.js, an image effects library powered by WebGL. It uses your graphics card for image effects that would be impossible to apply in real-time with JavaScript alone.

ProdEagle - Welcome

ProdEagle allows you to monitor your (App Engine) web applications in real-time using beautiful graphs and alerts without harming the performance of your application!

Workshop / Chrome Experiments

• dat.GUI: A lightweight graphical user interface for changing variables in JavaScript. Rather than modify variables manually in your code, use dat.GUI to modify them during execution.
• three.js: A lightweight 3D engine. Three.js makes it easy to render graphics with canvas, svg, and WebGL.
• stats.js: A JavaScript performance widget. Stats.js adds a simple info box to your page displaying frames per second (FPS) or, with a mouse click, the milliseconds required to render the frame.

Border Transition Effects

Bootstrap, from Twitter

Bootstrap is a toolkit from Twitter designed to kickstart development of webapps and sites. It includes base CSS and HTML for typography, forms, buttons, tables, grids, navigation, and more.

TameJS

Tame (or "TameJs") is an extension to JavaScript, written in JavaScript, that makes event programming easier to write, read, and edit. Tame is very easy to use in Node and other V8 projects

Glyphish

Icons for mobile apps

NounProject

The Noun Project collects, organizes and adds to the highly recognizable symbols that form the world's visual language, so we may share them in a fun and meaningful way.

Performance Data from our Fieldrunners WebGL Demo

Using custom events in Google Analytics to gather client-side performance data.

nowjs for Node

Directly call remote functions in Javascript: NowJS creates a magic namespace 'now', accessible by server and client. Functions and variables added to now are automatically synced, in real-time. Call client functions from the server and server functions from client.

SceneJS - WebGL Scene Graph Library

SceneJS is an open source 3D engine for JavaScript which provides a JSON-based API for defining and manipulating 3D scenes on WebGL.

Cocos2D JavaScript

Cocos2d-javascript is a 2D game/graphics engine based on cocos2d-iphone but designed to run in the web browser. It uses the latest features available in HTML 5 allowing real-time rendering of 2D graphics.

NClosure

Server-side Google Closure with Node.js. A fork of node-goog (which seems dormant).

Google App Engine Mini Profiler

gae_mini_profiler is a quick drop-in WSGI app that provides ubiquitous profiling of your existing GAE projects. It exposes both RPC statistics and standard profiling output for users of your choosing on your production site. Only requests coming from users of your choosing will be profiled, and others will not suffer any performance degradation.

SocketStream

SocketStream is a new full stack web framework built around the Single-page Application paradigm. It embraces websockets, in-memory datastores (Redis), and client-side rendering to provide an ultra-responsive experience.

Perceptual Image Difference Utility

PerceptualDiff is an image comparison utility that makes use of a computational model of the human visual system to compare two images.

Sitebricks

Sitebricks is a simple set of libraries for web applications. Sitebricks focuses on early error detection, low-footprint code, and fast development. Powered by Guice, it also balances idiomatic Java with an emphasis on concise code.

Apache Lucy

Apache Lucy is full-text search engine library written in C and targeted at dynamic languages. It is a 'loose C' port of Apache Lucene.

youtube-dl

Small command-line program to download videos from YouTube.com and a few more sites.

jiminoc/goose - GitHub

HTML content/article extractor

Scrollability

Native scrolling for mobile web apps... or at least the closest thing to it! Scrollability is a single script, it's small, and it has no external dependencies. Drop it into your page, add a few CSS classes to scrollable elements, and scroll away.

The Dynamics of Scrolling | MacResearch

"This got me wondering how difficult it would be to reproduce Apple’s momentum scrolling on your own in JavaScript. Is the reason no web developers mimic native scrolling that it is too difficult, or is it just laziness or the expectation that it is very difficult that stops them? Or is JavaScript just not up to the task? To find out, I decided to try. About 3 hours and 100 lines of JavaScript later, I have my answer."

Google App Engine Blog: Introducing ProtoRPC for writing App Engine Web Services in Python

ProtoRPC makes it easy to write consistent, reliable web interfaces that can be used, for example, to do the following:

• Create AJAX URL end-points for use by dynamic web pages and forms.
• Provide simple and understandable server-to-server communications.
• Become a back-end for command-line tools or other non-web based clients.

MTR

mtr combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool.

As mtr starts, it investigates the network connection between the host mtr runs on and a user-specified destination host. After it determines the address of each network hop between the machines, it sends a sequence ICMP ECHO requests to each one to determine the quality of the link to each machine. As it does this, it prints running statistics about each machine.

Crafty - JavaScript Game Engine

A lightweight, modular JavaScript game engine to easily produce high quality games. Includes a large variety of components such as animation, event management, redraw regions, collision detection, sprites and more.

MaxSquare: a fast and simple algorithm for maximizing photo thumbnail sizes

MaxSquare is an algorithm which calculates the optimal grid size for a collection of n images such that each image is allocated the largest possible square cell.

Coded Exposure Photography (Flutter Shutter Deblur, SIGGRAPH 2006)

In a conventional single-exposure photograph, moving objects or moving cameras cause motion blur. The exposure time defines a temporal box filter that smears the moving object across the image by convolution. This box filter destroys important high-frequency spatial details so that deblurring via deconvolution becomes an ill-posed problem.

Rather than leaving the shutter open for the entire exposure duration, we "flutter'' the camera's shutter open and closed during the chosen exposure time with a binary pseudo-random sequence. The flutter changes the box filter to a broad-band filter that preserves high-frequency spatial details in the blurred image and the corresponding deconvolution becomes a well-posed problem. We demonstrate that manually-specified point spread functions are sufficient for several challenging cases of motion-blur removal including extremely large motions, textured backgrounds and partial occluders.

Remote git diffs with bbdiff

1. See previous post about key exchange.
2. Add the following to .gitconfig:

   [diff]
       tool = "bbdiff"
   [difftool]
       prompt = false
   [difftool "bbdiff"]
       cmd = /home/mihaip/bin/bbdiff "$REMOTE" "$LOCAL"
   

3. Create bbdiff on the remote machine with the following contents:

   #!/bin/sh
   
   source_machine=${SSH_CLIENT%% *}
   for arg in "$@"
   do
     readlink -fn $arg
     if [[ $arg == /tmp/* ]]
     then
       scp $arg mihaip@$source_machine:$arg
     fi
     echo -n " "
   done | xargs ssh mihaip@$source_machine /usr/local/bin/bbdiff --wait --resume
   

Google Application Utilities for Python

Same set of utilities used to build and run internal Python apps at Google:

• Simple application startup integrated with python-gflags.
• Subcommands for command-line applications.
• Option to drop into pdb on uncaught exceptions.
• Helper functions for dealing with files.
• High-level profiling tools.
• Timezone-aware wrappers for datetime.datetime classes.
• Improved TestCase with many of the same methods as unittest2, plus helpful flags for test startup.
• google_test setuptools command for running tests.
• Helper module for creating application stubs.

xStats

xStats provides simple charts to help monitor your code performance.

• FPS: Frames rendered per second. The higher the number the better.
• MS: Milliseconds needed to render a frame. The lower the number the better.
• MEM: Megabytes of memory used. Make sure it doesn't keep incrementing.

Editing files remotely

1. See previous post about key exchange and get_source_machine
2. Install FUSE and SSHFS.
3. Ideally, mount the remote directories you're interested to the same path on your Mac. Most recently I've used:

   sshfs mihaip@snowberry.local:/D \
       /D \
       -oauto_cache,reconnect,volname=snowberry-mihaip
   

4. Add the following to your .bashrc:

   bbedit() {
     source_machine=$(get_source_machine)
     for arg in "$@"
     do
       readlink -fn $arg
       echo -n " "
     done | xargs ssh mihaip@$source_machine /usr/local/bin/bbedit
   }
   

Growl notifications from other machines

1. Install growlnotify
2. Set up public keys so that the other machine can SSH into the Mac without prompting
3. Add the following to .bashrc:

   get_source_machine() {
     echo ${SSH_CLIENT%% *}
   }
   
   function growlnotify() {
     if [ "$AT_HOME" = "true" ]; then return; fi
     source_machine=$(get_source_machine)
     ssh mihaip@$source_machine /usr/local/bin/growlnotify \
         -n "command-notifier" \
         -m \"[$PWD]\" \"$1\" 2> /dev/null
   }
   

4. In the Growl PrefPane, the notifications can be styled for the command-notifier "Application." (I like Music Video)

Using the Google V8 Javascript Engine in Java

V8 Java Bridge that allows any JavaScript code to run and call back to any provided Java methods.

JVM Crash in Development App Engine server after Mac OS X Java SE 6 Update 4

Restoring the previous JDK (from Time Machine) to /System/Library/Java/JavaVirtualMachines/1.6.0.3.jdk and pointing Eclipse at it worked for me.

Backbone.js

Backbone supplies structure to JavaScript-heavy applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing application over a RESTful JSON interface.

Underscore.js

Underscore is a utility-belt library for JavaScript that provides a lot of the functional programming support that you would expect in Prototype.js (or Ruby), but without extending any of the built-in JavaScript objects. It's the tie to go along with jQuery's tux.

Underscore provides 60-odd functions that support both the usual functional suspects: map, select, invoke — as well as more specialized helpers: function binding, javascript templating, deep equality testing, and so on. It delegates to built-in functions, if present, so modern browsers will use the native implementations of forEach, map, reduce, filter, every, some and indexOf.

CoffeeScript

CoffeeScript is a little language that compiles into JavaScript. Underneath all of those embarrassing braces and semicolons, JavaScript has always had a gorgeous object model at its heart. CoffeeScript is an attempt to expose the good parts of JavaScript in a simple way.

The golden rule of CoffeeScript is: 'It's just JavaScript'. The code compiles one-to-one into the equivalent JS, and there is no interpretation at runtime. You can use any existing JavaScript library seamlessly (and vice-versa). The compiled output is readable and pretty-printed, passes through JavaScript Lint without warnings, will work in every JavaScript implementation, and tends to run as fast or faster than the equivalent handwritten JavaScript.

d3.js

D3 allows you to bind arbitrary data to a Document Object Model (DOM), and then apply data-driven transformations to the document. As a trivial example, you can use D3 to generate a basic HTML table from an array of numbers. Or, use the same data to create an interactive SVG bar chart with smooth transitions and interaction.

iOS UI Patterns

Chef - Opscode

Chef is an open source systems integration framework built to bring the benefits of configuration management to your entire infrastructure.

boilerpipe

The boilerpipe library provides algorithms to detect and remove the surplus 'clutter' (boilerplate, templates) around the main textual content of a web page.

Instance/web API running at http://boilerpipe-web.appspot.com/.

visionmedia/express - GitHub

Fast (and small) server-side JavaScript web development framework built on node and Connect.

Varnish: Notes from the Architect

"I have spent many years working on the FreeBSD kernel, and only rarely did I venture into userland programming, but when I had occation to do so, I invariably found that people programmed like it was still 1975."

Watir

"Watir, pronounced water, is an open-source (BSD) family of Ruby libraries for automating web browsers. It allows you to write tests that are easy to read and maintain. It is simple and flexible."

Used by Facebook.

Graphite - Enterprise Scalable Realtime Graphing

Graphite is a highly scalable real-time graphing system. As a user, you write an application that collects numeric time-series data that you are interested in graphing, and send it to Graphite's processing backend, carbon, which stores the data in Graphite's specialized database. The data can then be visualized through Graphite's web interfaces.

Used by Etsy.

Etsy also has StatsD (blog post) which is a daemon that increments/times can be sent to, which aggregates them and sends them to Graphite.

ack 1.94 -- better than grep, a source code search tool for programmers

"ack is a tool like grep, designed for programmers with large trees of heterogeneous source code."

Unicode code converter

Convert input text to various representations and escaping levels.

Also related: Text Escaping and Unescaping in JavaScript.

List of JavasScript Game Engines

List of JS Game Engines (community effort) : javascript

sam-mccall/node-plate

Syntactic sugar for running async functions in sequence.

Also Do.

web-page-replay

"Record web pages using a local DNS and a local web server which act as proxies and records performance metrics in addition to the actual resources."

Node Inspector

dannycoates/node-inspector - GitHub: "Node Inspector is a debugger interface for nodeJS using the WebKit Web Inspector."

Introducing Knockout, a UI library for JavaScript

Knockout is a JavaScript library that makes it easier to create rich, desktop-like user interfaces with JavaScript and HTML, using observers to make your UI automatically stay in sync with an underlying data model. It works particularly well with the MVVM pattern, offering declarative bindings somewhat like Silverlight but without the browser plugin.

CSS Timing Bookmarklet

This bookmarklet loads this script which uses querySelectorAll on all the selectors in loaded stylesheets and then sorts their eval time.

DNode

substack's dnode at master - GitHub: "DNode lets you expose a JavaScript function so that it can be called from another machine using a simple JSON-based network protocol. That’s relatively straight-forward... but DNode is designed for asynchronous environments, and so also lets you pass callback functions which will be translated in to references and used to make remote method invocations back to your original client. And to top it off, there’s a browser client library so you can perform the same trick over a WebSocket between a browser and a server."

creationix's step

A simple control-flow library for node.JS that makes parallel execution, serial execution, and error handling painless.

JSRegexTeststand v2

Simple single-page app that assists in dynamically developing and testing JavaScript regular expressions. Think of it as a REPL for JavaScript regexes.

jsctags

ctags-compatible code indexing solution for JavaScript

Learn CSS Positioning in Ten Steps: position static relative absolute float

Awesome concise explanation of CSS positioning

Code Like a Pythonista: Idiomatic Python

WebKit Internals

• WebKit Page Cache I – The Basics
• WebKit Page Cache II – The unload Event
• Ruby Rendering in WebKit
• How WebKit Loads a Web Page

Twurl

curl-like tool for the Twitter API

And oacurl is a more general OAuth curl-like tool

Bit.ly command line fu

Akihabara

Set of libraries, tools and presets to create pixelated indie-style 8/16-bit era games in Javascript.

Node.js

Evented I/O for V8 JavaScript.

Jetty

http://codeshal.tumblr.com/post/203700970/jetty-embedded-mode
http://wiki.eclipse.org/Jetty/Tutorial/Embedding_Jetty
http://wiki.eclipse.org/Jetty/Reference/Dependencies#Dependency_Tree
http://wiki.eclipse.org/Jetty/Starting/Downloads

Easy Performance Profiling with Appstats

Easy Performance Profiling with Appstats

App Engine JavaScript SDK

"App Engine for JavaScript! With App Engine, you can build web applications using the JavaScript programming language, and take advantage of the many libraries, tools and frameworks for JavaScript that professional developers use to build world-class web applications. Your JavaScript application runs on Google's scalable infrastructure on top of Java and Rhino, and uses large-scale persistent storage and services."

Recipe: CloudKit

"CloudKit is Jon Crosby’s neato web app that vends schemaless-but-versioned JSON objects via a RESTful web API."

objectify-appengine - Project Hosting on Google Code

"Objectify-Appengine provides the thinnest convenient layer which addresses these issues, yet preserves the elegance of get, put, delete, and query."

MongoDB

JSON key-value store

Using a Caching Proxy Server for Web Demos

Simple python caching proxy server: run your test once against the proxy server in 'record' mode, and then after that you can use the server for consistent local playback.

Super Awesome Buttons with CSS3 and RGBA

Mercurial: The Definitive Guide

Alternative Mercurial guide.

Hg Init: a Mercurial tutorial by Joel Spolsky

For when I want to actually learn a DVCS:

Mercurial is a modern, open source, distributed version control system, and a compelling upgrade from older systems like Subversion. In this user-friendly, six-part tutorial, Joel Spolsky teaches you the key concepts.

Gecko Memory Profiler

Add-on appears to be stuck at Firefox 3.5-only though.

Labs/Memory Profiler - MozillaWiki: "Memory Profiler is an experimental Labs project that attempts to make debugging memory use in web pages easier.

For more information about the tool's design and rationale, see Atul's blog post entitled Web Application Memory Profiling, Take Two."