Web app for pedals

[balazs.bencs]

Hey folks!

As a web developer, I was experimenting with some big data and web crawling robots, and I got an idea. I was thinking about having a web app (and/or a mobile app), which would contain all the pedals from this site in a more searchable format, a proper database with a simple and clean web UI, with all the existing data already in it.

So far I managed to create a simple web crawler which crawls through the site based on the paginator on the bottom, and saves every pedal's information (the name of the pedal, the image URLs (downloading them too), and the number of comments into a simple MongoDB database), mostly because I was curious which pedals were the most "popular" (based on the number of comments).

This was a simple experiment so far, but I figured, that I could continue working on this, and save tags, and comments, creation of date, etc.. all the useful data, and build an API and a web based UI for it, and make it open source on github. This could be a much more useful and searchable database, for everyone I think.

My question is, do you see a potential in a system like this? We can figure out the details later together of course, I just wanted to throw the idea out there, and get some opinions and ideas and see if worth to create a system something like this, so please share your thoughts.

[Marbles]

That would be amazing!

How far do you want to take it? :D

I sometimes have trouble finding effects cause theyre basically a copy of another effect. I scroll through all fuzzes in the hope of finding it (which is good cause I come across more awesomness), but maybe something like related effects, or maybe family? As in; Fuzz Face family, Tonebender, fuzzrite, tubescreamer etc.


What would've come in handy when I didn't have a lot of parts, was to search for key-component. As in: I have a few BC109c's , what effects can I make with that/ have that? Or: I have a JRC4558 and these value pots, what can I make with those.

The question you can ask ofcourse is: A lot of these factors are a matter of taste and can be changed, but for people wanting the stock values it would be awesome.

[Goga]

Hellooo!

Excellent Idea my friend and also very very enthusiastic.

This blog is made by Mark and he has to tell the final decision
and the admins too.

I will tell everyone something, you all can judge me for that but we have to see that side of the story too.
If it's an app the Admins and You should take in mind that it shouldn't be for free.
Just because everything will be served on the table as we say. Paying for the app will be superb opportunity for the creators to get some feedback from their enormous efforts though the years and also will fond this blog for further. Other Idea for the app is- Limited quantity of free project and symbolic prices for the ones that you want to download. With this symbolically will pay respect for the time and efforts of the people creating the projects. Third idea is- Getting sponsored by the companies that make the original stuff and promoting them trough the app. Etc...

And so far and so on! I like you all and I like the idea!!!!!
Support
Goga    

[balazs.bencs]

These ideas are great! Actually, I started thinking about the parts stuff, it would be cool to have a filter based on components, but also would be hard to write an algorithm to identify the existing veros to see what components do they contain. This certainly needs a human eye I think - that means a LOT of work.

Also, I had the idea to have some sort of substitute section, where you could easily find alternative components for the common parts that we all use.

[balazs.bencs]

Goga,

I have to disagree with you, because this site is all about sharing a lots of pedal schematics, sharing knowledge, helping other people.
If I (or we) start this project, the only way to go is in my opinion is to build it with a open source licence and share everything that we've done - so that other people can learn from it, or make it better by contributing to the project. This project should be definitely a community effort I think. But you are right, first I would really love to hear Mark's opinion about this.

[Marbles]

Ofcourse,  you're right. First of all, let me add that I actually love this site and don't need it to change whatsoever. I'm just a visitor and it's just ideas. Mark and the other guys obviously call the shots :)
Paying for things and make it a commercial thing is not something that I would be rooting for, but maybe I misunderstood.

If something like searching by component would/could be added, ofcourse it's not possible to recognize parts from the veros, but I wouldn't mind personally to registrate each one by one.  It's a shitty job, but would be a way for me to contribute to all the things this community gave me. Just repaying. But you know... 'if' ;)

[rocket88]

I can assure you there is no intention of any profiting, commercializing, or sponsoring of the site. So no worries about having to pay for anything.

[scimitar]

Seems a fantastic idea and I agree that being able to search on components would be very useful, it would be nice to be able to generate statistics about which components and values are most common as it would help in stocking up on common parts. I would also be prepared to put some time into populating these component lists as a way of giving back.

[Pavlos]

I fully agree with the site staying open and free to use for all, it is one of the great strengths of the site that it is so easy to look, to learn, to ask questions, to look and learn some more, to ask even more questions......and at some point be able to answer someone elses questions and help pass on the knowledge so freely shared by others. We have a great community here of people from all over the world, of all ages, abilities and any other way you care to differentiate us, but one thing that we all have in common is that we want to to have a go at building our own stuff, and are willing to share time, experience, ideas and the efforts of our labour with each other. A big thank you to everone who contributes to the site in any way no matter how small and no matter the level you are at, it's all good  

balazs.bencs, very generous offer of your skills and time
The idea about being able to generate statistics for components is an interesting one, perhaps with a breakdown of the type of circuit would be useful as well if possible? especially for those new to building. After all knowing that component A is used in a load of circuits is potentially misleading if it is used mostly in fuzz pedals and you are wanting to build delays and phasers it's definitely a good idea though, and I'm sure it would still be helpful to get a broad overview of how widely used a particular component is and in what kind of circuits

[notnews]

a few thoughts:

(tl/dr: a wikipedia-esque effects pedal DIY database where users can contribute to content that a small group of admins can approve before the content goes "live")

I am SUPER behind this idea... to the point where I would love to contribute time and effort towards making it right and making it full.

I would like to see a centralized database, where each effects pedal's "page" includes:
- its lineage (if it's based on another circuit or is in a certain family of circuits)
- a schematic, a vero, a pcb, component/parts list, pictures of the effect
- user reviews of the effect
- place to have discussions about the effect... experiences trying to build it up, issues with debugging, etc (OR, discussions can be restricted to this forum, which may be more neat anyway)

regarding a component-based engine... you're certainly right that having all of the information in the searchable database upon start-up would be a monumental effort. BUT, what about a crowd-sourced info platform? Something akin to Wikipedia? So for example, let's say the site starts up with a schematic, vero, pictures, reviews, etc for the Dwarfcraft Shiva... but it's missing it's component list. As a visitor, I could navigate to the Shiva's page and add the component list myself. Then, one of the site admins could perform the simple(r) task of just reviewing the info I'm trying to post/submit to the site... which at that point would just be cross-referencing the vero/schematic to the parts list (a feat easier than creating the parts list from the vero/schematic itself).

I strongly urge the creators/initiators of this database to include the large pool of tagboardeffects users into their project. We can help make the site great... we all care about effects, to the point where a good number of us would gladly take time to carefully and diligently put organized information into the database. It would be a shame to ignore that resource, because I think it can be trusted... and like I said earlier, can also have a requirement to be verified by a smaller group of admins or something.

edit: I just read above that Marbles had the same idea, asking if he can contribute to the database by entering component names/types/values for a given pedal. great idea!

[Muadzin]

If anyone should be funding this app it should be the likes of Tayda, Smallbear or Mouser. Because the constant stream of new layouts means constant 'I want to build this!' followed by 'I need to order parts for this'. So if anyone has profited financially from Tagboard its those companies.

[motterpaul]

I have been saying for YEARS that the parts for each build need to be listed in text on each page. Currently the parts are all listed only within the images, and there is no way at all to search for builds based upon what parts they contain.

I know this is not a simple fix, because it is very hard to scan images for words (impossible?), but if the parts could be added to a searchable database for builds that would be another way to accomplish the same thing.

Great idea - I hope you make it work. It would be a big help to many of us and greatly appreciated by me, for sure (speaking just for myself).

[scimitar]

Did anything ever come of this, I know I for one would be willing to take part in the data gathering part (generating the BOMs) even if only to create a searchable database of builds vs. components. All we would need to get started is agreement on a database structure.

[Marbles]

Yeah, I'd still be up for it too. If I'd know a format I should put it into, we could just do a category each or something like that. But yeah, I have no idea how o add a database like that or how the collected data could be 'collected' best.

[balazs.bencs]

Sorry guys, I haven't forgot, I started to create a REST API as a base, but I'm leading R&D projects with machine learning now, and I have to study a LOT nowadays. I'll get there eventually ;)

[Marbles]

Haha, understandable.
Good luck with that! If in the future you need some data harvesting monkey workers let me know!

[balazs.bencs]

Hey guys,

I've made some progress on this. I was able to crawl the entire site with a Python script, and saved all the data, so I have some data to start work with. I only saved the title of the posts, the description and the images.

This week I'll import it to the database I designed, and share the REST API that I built to be able to query the database. (I have to deploy to the cloud somewhere)
This whole thing is built with Loopback (IBM's NodeJS based REST API solution), it will have some docs generated so that everyone would be able to use it.

Next up I am going to create a small React app so that we could use this through a web UI - I want a simple UI for trying out the system, and we'll see how things will go. It is going to be a long journey, but finally, I will be able to make it happen and everyone will be able to use it and give feedback.

Oh, and because I'm doing it with React, we'll have a chance to create an app for Android and iOS too (React Native is cool!).

[Marbles]

Great work!

I only understand about half of what you're saying, but sounds great! :D

Keep u posted about the progress!

[scimitar]

I know we are almost 18 months on from the last post, so I suspect I know the answer, but did anything ever come out of this, I am increasingly frustrated that I have certain components that I would like to use but can't find what layouts use them - and I don't have the patience or time to trawl through all of them?

[balazs.bencs]

actually I started working on it not long ago, and I have a beta version with an admin interface half-done.

Soon, I hope I can show you guys something.