Hackpads are smart collaborative documents. .
976 days ago
Unfiled. Edited by Lyre Calliope 976 days ago
Following Friday's Citizen's Rising symposium, this workshop continues the Anti-corruption weekend in which we learn about and mobilize around how to get big money out of politics!

976 days ago
Unfiled. Edited by Scott Linford 976 days ago
Scott L Changes - Sept 23, 2014
Added ZIP functionality to both scrape.py (Independent) and scrape2.py (Lobbyists). 
  • Files are downloaded into folder
  • An archive is created. The entire folder is ZIPed up.
  • Code has been committed to GitHub
Original Prototype Presented - Sunday, Sept 21, 2014
Working Python prototype. Downloads all PDFs listed at:
 
976 days ago
Unfiled. Edited by Scott Linford 976 days ago
Scrape Tool Discussion - issues, suggestions, future direction
Scott L Handle IO Errors
The NH website is flakey. IO errors are common. A single IO glitch causes the Scrape Tool to crash leaving behind an incomplete set of reports. The tool must then be restarted by hand from the beginning. There is no easy way to continue from where it crashed. Retry logic is a better solution, easier to implement, easier to test. Retry logic would prevent the script from bombing out. 
Add error handling logic:
  • Retry logic for loading HTML
  • Retry logic for downloading PDF
  • Skip item after n retries
  • log the failed URL
  • move on to the next report
 
976 days ago
Unfiled. Edited by Scott Linford 976 days ago
  1. ensure that the scrape tool can be run periodically, potentially by a novice computer user (" 1-click trigger?)
 
976 days ago
977 days ago
Unfiled. Edited by Scott Linford 977 days ago
Scraping 2013 Lobbyist Reports
 
Scott L Working Python prototype. Downloads all PDFs listed at:
  • <etc>
Code posted on Github
  • (I know, silly name. Needs some reorg)
Run from the command line
  • python scrape2.py
Logic was nearly identical to Scraping 2012 Independent Expenditures. This script new one just does it over and over. The sections are read dynamically. 
 
I have not run it to completion yet. It takes a while. Further testing ahead in other words.
 
978 days ago
Unfiled. Edited by Chris Catterton 978 days ago
Chris C Politricks
 
People (Name, contact info, skills, etc)
Team Lead: Chris Catterton - chris@catterton.net
Team Members:
-Adam Friedman 
-Brian Boyko
 
General Project info
What question(s) is this project is trying to answer?
-how can we give interested citizens the best high-level toolkit to get involved?
-how can we help the different members of the anti-corruption movement work together and share content?
 
Relevant links (github repo, deployment url, etc)
 
Current status
Strikingly page up & running
  • Features an initial list of high-quality media, a comments section for user submissions, and an index of organizations involved in the movement for a government that is more responsive to the concerns of ordinary citizens{
 
Next steps
Community interest?
Partner organizations?
Better design? (Strikingly = quick & dirty)
 
Members (56)
Jacob Turino Scott Linford darius@clikburn.com Steve Rand Presley Pizzo Niki Borofsky Chris Catterton Jacob Brennan Michael McCarthy Christopher Scranton Hai Jung Theresa Kim Keller Scholl Jack Gray Carla Miller Jacob Cole Daniel Porter Devers Talmage Kostas Koutsioumpas nadeem@nimblebot.com cmiller@ethics.harvard.edfu
Collections

Create a New Collection

Cancel

Move XXX to XXX


XXX will be invited to the XXX on XXX.

Cancel

Contact Support



Please check out our How-to Guide and FAQ first to see if your question is already answered! :)

If you have a feature request, please add it to this pad. Thanks!


Log in