Browse › Data and git scraping
Data and git scraping
Git keeps every version of a file, so a dataset in git has its whole history. "Git scraping" uses that on purpose: a scheduled job fetches some data and commits it, and the log becomes a record of how it changed.
How people use git for it
- Keep data as CSV or JSON so changes show line by line.
- A job fetches the data on a schedule and commits only when something changed.
- Anyone can see exactly when a number changed, and to what.
On GitStable
Nothing public here yet. Projects are sorted into categories from their files when they're pushed, and owners can change it in Settings.
Create an account and push yours, and it can be the first.
Examples hosted elsewhere
- Git scrapingsimonwillison.net/2020/Oct/9/git-scrapi…
The write-up that named the technique.
- ca-fires-historygithub.com/simonw/ca-fires-history
California fire data, committed as it changes.
- Our World in Data's COVID-19 datagithub.com/owid/covid-19-data
A public dataset kept in git.
Other things people do with git
Apps and tools 26Libraries and packages 11GamesWebsites 4Plugins and extensions 1AI prompts, skills and agents 1Documentation 1Awesome listsBooks and writingNotes and knowledge basesSettings files (dotfiles)Infrastructure as codeScience and notebooksTeaching and coursesHardware and 3D printingFonts, icons and design