Get best of the week

Museria - a decentralized repository of music

image

I once gathered to write an application to select music for myself and listen at home / on the street / training, etc. And so that all this works in a stream, with a minimum of my participation. He came up with the architecture, sketched a prototype and ended up with one “small problem”.

And it is not clear where to get the song files themselves. By this moment, VKontakte had already closed its api, everything was deaf on large music portals, even songs were given in pieces so as not to be parsed. There were only some separate one-day websites with tons of advertising and all kinds of garbage, all kinds of dubious grabber programs and other “dirty” options. In general, not a single valid good decision. You can, of course, buy a subscription to some Yandex music or the like. But then again, there is nowhere open public api and you do not have access to music programmatically. Several large companies, in fact, restricted the rest of their access to music. Why did this happen at all? Digging deeper, it became clear that the main problem in copyright. The current solution in the form of subscriptions suits many commercial authors of musical works and these same companies.At the same time, non-profit and shareware music also fall into the general list. You either pay for everything or you don’t listen to anything at all.

And I began to think what to do with all this. How can free music distribution be organized? What would I do if I was creating music myself and would like to make money on it? Would I like it if my songs began to be distributed in a pirated way? What is an alternative solution?

As a result, there were two main problems that need to be addressed:

  • Organization of free distribution of music by methods convenient for most people, including software.
  • Offer alternatives to music creators for money

Global decentralized music repository


Initially, I tried to find existing solutions and create everything based on this. After some time searching, ipfs first liked it . I started implementing my idea, but after a while I found several critical problems in this solution:

  • Ipfs - storage for everything and everything. Here, and images and music and video, and anything else. In general, such a big planetary "garbage". Therefore, when you start your node, you immediately get a huge load. The car just writhes in pain.
  • - «». , , , , . , . , , ipfs .
  • ( ), . , , . , , , . .

There were still many minor problems, the impression was then unambiguous: this cannot be used for the project. I continued searching for the repository, explored various options, but still did not find anything suitable.

In the end, I decided it was worth trying to write a decentralized storage myself. Let it not pretend to be an interplanetary one, but it will solve a specific task.

This is how spreadable , storacle , metastocle , museria , museria-global turned out .

spreadable- This is the main, the lowest layer, which allows you to combine nodes into a network. It contains an algorithm that I have so far partially implemented based on somewhere in 10,000 servers. The full version of the algorithm is much more difficult to implement and would require a few additional months (maybe more).

In detail, I will not describe the spreadable in this article, I’d better write a separate one somehow. Here I just note some features:

  • It works through http / https.
  • You can create a separate network for a specific task, which will significantly reduce the load on each individual project than if they were all on the same network.
  • Initially, a mechanism with timeouts and other little things was thought out. And this works for all methods in both the client and the node. You can flexibly manage settings from your application.
  • nodejs. . «», . : , , , ...

storacle is a layer inherited from spreadable that allows you to store files on the network. Each file has its own hash according to the contents, by which it can be obtained later. Files are not divided into blocks, but are stored in their entirety.

metastocle - a layer inherited from spreadable, which allows you to store data on the network, but not files. The interface is similar to the nosql database. You can, for example, add a file to storacle, get its hash and write it to metastocle with reference to something.

museria - inherited from storacle and metastocle. This layer is directly responsible for storing music. The repository only works with mp3 files and id3 tags.

As the "key" to the song, its full name is used in the form Artist (TPE1) - Title (TIT2) . For instance:

  • Brimstone — The Burden
  • Hi-rez — Lost My Way (feat. Emilio Rojas, Dani Devinci)

You can find out as much as possible how song names are formed here . We need to look at the utils.beautifySongTitle () function .

A key match is a percentage defined in the node settings. For example, a value of 0.85 means that if the key comparison function (song names) found a similarity of more than 85%, then this is the same song.

An algorithm for determining similarity is in the same place, in the utils.getSongSimilarity () function .

Cover to the song, for later retrieval, also attached via tags ( APIC ). The utils have all the necessary methods for receiving and processing tags.

An example of working with the repository through the client can be found in readme .
. , .

museria-global is an already configured git repository for launching its own node in the global music network. Clone npm i && npm start and that's it. You can configure in more detail, run in docker, etc. Details are on the github .

When the repository is updated, you need to update your site. If the major or minor version number changes, then this action is required, otherwise the old nodes will be ignored by the network.

You can work with songs manually and programmatically. Each node starts the server for various tasks. Including, when you visit the default endpoint, you will get an interface for working with music. For example, you can go to the root node(the link may not be relevant later, the input nodes can also be obtained by telegram , or see updates on the github).

So you can search and upload songs to the repository. Downloading songs can take place in two modes: normal and moderated. The second mode means that the work is carried out by a person, not a program. And if you check this box when adding, then you will need to solve the captcha. Songs can be added with priorities -1, 0 or 1. Priority 1 can only be set in moderated mode. Priorities are needed to make the repository more effective in deciding what to do when you try to replace an existing song with a new one. The higher the priority, the more likely you are to overwrite the existing file. This helps fight spam and increases the quality of downloaded songs.

If you start adding songs to the repository, try to attach images (cover), although this field is optional. In 99% of cases, the very first pictures in Google by song names are album covers.

How technically files are added in a nutshell:

  • The client receives the address of a free node, which for some time will become the coordinator.
  • The function of adding a song is triggered (by a person or a code), a request is made to add a coordinator to the endpoint.
  • The coordinator calculates how many duplicates need to be saved (configurable parameter).
  • The most suitable nodes to save are searched.
  • The file directly goes to these nodes.

How technically the files are received:
  • The client receives the address of a free node, which for some time will become the coordinator.
  • The function of receiving the song is triggered (by a person or a code), a request is received to get a coordinator to the endpoint.
  • . , , .
  • , .


I have always been interested in the question, how can one objectively evaluate the value of many creative works? Why, for example, does a person expose his music album for $ 10? Or for $ 20 or for $ 100. Where is the algorithm? When, for example, we are talking about some kind of physical product, or even many types of services, then we can at least calculate the cost price and proceed from this.

Ok, let's put $ 10. Is it really effective? Suppose I listened to an album somewhere or a song from there and decided to thank. But according to my feelings and my own abilities, $ 3 is my ceiling. And how to be here? Most likely I just will not do anything, like most people.

By setting a fixed price for creative work, you simply limit yourself, do not let a large number of people send you less money, which in total can be more impressive than those who buy at the price you set. It seems to me that creativity is exactly the area where donations should first steer. To do this, you need:

  • To teach people to give thanks in this way. The creators themselves must clearly show that they would like to receive donations, add everywhere links to different payment methods, etc.
  • . , , .

    , :

    http://someartistsdonationsite.site/category/artist?external-info

    , :

    http://someartistsdonationsite.com/music/miyagi?song=blabla

    .

    museria , , / . . , , .

, , ,


  • , , , . museria , . , , , .
  • - : , , ...
  • , . , .
  • - . , , .

The project is now in its infancy. A test network is running, nodes can often reboot, require updates, etc. In the absence of critical problems during the evaluation period, the same network is transformed into the main one.

View information about the site from the outside: the number of songs, free space, etc., you can click on the link type http://node-address/statusor My contacts:http://node-address/status?pretty



More articles:


All Articles