TO Database/Manager

toutils

New Member
Joined
Mar 23, 2017
Messages
4
Reaction score
4
Points
203
Age
41
Gender
Male
The TO database manager (todbmanager) is now released along with the TO database.
The code can be found here:
https://github.com/toutils/todbmanager

The database and tagged releases are here:
https://github.com/toutils/todbmanager/releases

The turkopticon thread on it is here:
(old thread)
https://turkopticon.ucsd.edu/forum/show_post/1533
(new thread)
https://turkopticon.ucsd.edu/forum/show_post/1813

It's written in python and puts the database in an sqlite file. Once it has a database, it can autoupdate, setting order by edit date and going back only as far as its getting new data.

On first run it gives you the option to download the database from the wiki, and use that to auto-update from, to avoid having to do a full scrape.
(this is disabled at the moment, to use the database with the scraper and only do updates, download the database from the releases tab, extract it to the directory you extracted todbmanager, and rename the database to to.db)

Any comments or suggestions are welcome, I'll keep this thread up to date with developments.

If you would like your turkopticon id to be added to the blocked ids list, add "[NODB]" (without the quotes) to any one of your turkopticon reviews. It can be anywhere and it doesn't have to be in all of the reviews, just one. It will be picked up the next time I do a scrape. If your reviews are still in the database at that point send me a message on here, post in the TO thread, or open an issue on github.

Note for requesters: Requests to have your requester id filtered from released databases will be taken on a case by case basis. I won't filter out your requester id simply because you have a bunch of bad reviews. It's up to the individual user to decide weather or not they want to respect these filtering lists for their own databases.

At the moment it doesn't scrape for hidden reviews, and the database doesn't contain them. This requires a separate full scrape, along with a full database, to hit requester ids individually. This will be released in a future update.

Edited to be up to date.
 
Last edited:
  • Like
Reactions: Kadauchi

toutils

New Member
Joined
Mar 23, 2017
Messages
4
Reaction score
4
Points
203
Age
41
Gender
Male
The initial post above will be edited for updates as they come.
 
  • Like
Reactions: Kadauchi