Cross Municipality API

[hawkticehurst]

Idea / Feature

Create an API that enables queries across multiple CDP municipalities.

Use Case / User Story

As a researcher, I want to be able to easily gather data from across different CDP municipalities so that I can ask/answer questions about local politics across different regions.

Todo: Insert more user stories

Other Notes

I've only given a user story that frames this API as helpful to researchers, but an API like this could have incredibly wide applications.

I could envision a really cool section of our public website where we conduct some basic data analysis and present that data in an interesting / visually compelling way.

I also suspect there could be a multitude of new product features that would stem from this one API.

We could also even go the route of hosting this data as an open-source API that could be used by anyone for cool projects.

[hawkticehurst]

cc @JacksonMaxfield for more thoughts

[evamaxfield]

I have a user story:

"I want to see how many municipalities discussed 'housing and homelessness' in the last month and which municipalities brought legislation regarding 'housing and homelessness' -- further, what was the general sentiment during discussion of the topic?"

This is an interesting one because I think there are Python API design things we can do to make this easily possible and there is a larger project here about making a website for such queries.

This is where plotting and visualization would come in handy. A website to interactively answer these questions and not just a Python API for people like me.

---

Big note: this idea requires multiple municipalities before work even begins imo. This is a really big project but something that we should definitely head towards imo.

[hawkticehurst]

Ooh, I love the idea of having a website/webpage (maybe data.councildataproject.org or visualize.councildataproject.org??) that provides an interactive way to plot, visualize, manipulate, etc the data that would be accessible from this API.

[evamaxfield]

@nniiicc added some comments to a new issue (#19) that I am transferring here:

Idea / Feature

Support programmatic use of CDP data

Use Case / User Story

• I am a political scientist. I want to query a CDP API so that I can obtain all transcripts from city councils in the USA where the > phrase "critical race theory" appears.
• I am a political scientist. I want to query a CDP interface so that I can obtain all transcripts from city councils in the USA where > the phrase "critical race theory" appears. I want the interface to allow me to select criteria like a a date range, enter a keywords, > choose the format of delivery etc.

Solution

• A documented restfulAPI with examples how to query within and across CDP instances
• A GUI that allows a user to assemble and download a dataset

Alternatives

• Pre-packaging / releasing CDP data at regular intervals. We could just say here is a giant tarball of all transcripts as a first step, but ultimately this is a poor solution to encouraging more programmatic use of CDP data.

Stakeholders

My use cases are for research - but it could be useful to lots of people from lots of different backgrounds (computational > journalism, advocacy group, etc) that want to use CDP data

---

From a while back, this was somewhat waiting on CouncilDataProject/cdp-backend#106 which was just resolved and released in cdp-backend v3.0.0.dev19.

Although, we are stilling waiting on multiple municipalities to be deployed / Seattle redeployed.

I can get started on cdp-data or some Python package that specifically aims to assist in dataset creation and processing now!

On to the other asks from Nic. A GUI that allows a user to assemble and download a dataset. I think this was the idea because data.councildataproject.org

---

On a different note, there is enough of a problem now that I think I should have a package called cdp-instances that simply stores a bunch of JSON objects for the various CDP instance details. It would be useful to pull in both in Python for this project and for JS for use in rendering all the instances we have available on the main CDP website.

That should be easy for me to do (and it's simple enough that we may use it as a model for CouncilDataProject/cdp-backend#167.

---

Last comment, I think I should add a database table to CDP instances firestore that store metadata about the instance. I.e. backend version... Will look into it. But it would be useful to have for both maintaining the instance and for mass querying.

I.e. when constructing a dataset, if a feature was only introduced in v3.2 of CDP, then ignore all instances that aren't on v3.2